Art in Noise: an Embodied Simulation Account of Cinematic Sound Design

Home , Carl Lange (physician), Cecelia Hall, Come to Nothing, Mark Mangini, Richard Beggs, Rinko Kikuchi

Art in noise: an embodied simulation account of

cinematic sound design

Mark Shelton Ward

A thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy

Supervisor: Professor Ross Gibson Faculty of Arts and Design University of Canberra Australian Capital Territory, Australia

July 2017 Emendations completed 21 April 2020

Abstract

Sound and moving pictures share over a century of entwined history yet the dominant visual narrative accounts of cinematic sound lack explanatory power. This thesis responds by developing a novel conceptual framework using the paradigm of embodied cognition. An embodied cognition approach provides a means to analyse the functional architecture, meaning-making capacities, and aesthetics of cinematic media in ways not afforded by conventional media theory.

The framework deploys embodied simulation theory (Gallese 2005) to emphasise the affective aspects of Feeling of Body (Wojciehowski & Gallese 2011) and puts into play the concept of embodied meaning (Johnson, M 2007b) which asserts all human meaning and abstract conceptual thinking have basis in our sensorimotor interactions with the world. In the context of cinematic media, embodied simulation theory suggests the stylistic crafting of sensory-perceptual content not only elicits affect but shapes cognition. Specifically, a four-tiered conceptual framework is advanced using embodied simulation theory to contain, integrate and structure three extant models for the elicitation or representation of affect at discrete levels of description. A strength of the four-tiered framework is a highly granular and scalable capacity to describe how cinematic sound may produce embodied meaning.

To illustrate the explanatory power of an embodied simulation account of cinematic media, I examine the design strategies used in my own cinematic sound work – specifically, Jane Campion’s In the Cut (2003) – and the work of other sound designers.

Lastly, the implications of an embodied simulation account of cinematic media are discussed in relation to investigating the body-mind nexus, bridging the cognitive sciences and Cultural Studies, and informing institutions involved in media education and cultural policy.

To Prue, without you I would have stumbled, and never got up.

My deepest gratitude goes to my principal supervisor, Professor Ross Gibson, whose patience allowed me to explore disciplines far flung from my native Film Studies. Without his encouragement and quiet intelligence I would have abandoned my quixotic quest a very long time ago.

Research is a solitary and oftentimes isolating pursuit; but along the way I met many people whose wisdom and generosity I encountered at just the right time. Space does not permit me to acknowledge all their names, but some demand it. Deserving of special mention are Ed Tan (who showed me how a humanities scholar might use science to reveal hidden truths), and the late Nico Frijda (whom I met all-too-briefly but whose candid advice encouraged me greatly).

Another deserving of special mention is the late Andrew Plain, a big-hearted man and a gifted sound designer who allowed me space to make some noise.

To my teachers of media, I acknowledge a great debt. To the late Bill Constable, who opened a door and pushed me through it. To Noel Bolden, Barry Fernandes and Noelig Quinn, my shrewd shepherds while a student at AFTRS in the 1980s, always ready to help, but only when really needed. And to Kari Hanet, who first opened my eyes to the provocations of film theory.

I also thank my own students of media, especially those during my time as Head of Sound at AFTRS (2006–2009), who unwittingly put my ideas to the acid test.

There is nothing quite like the push-and-shove of industry to sort the theoretical wheat from the chaff, and many of the thorny issues which motivate this research I encountered during my early experiences in industry. And so I am grateful for the time I shared with David White, Victor Gentile, and Liam Egan, my colleagues in the founding of Counterpoint Sound and Oracle Pictures; together we grappled with just

I also acknowledge the skills of professional editor Deborah White who provided proofreading services according to the university-endorsed ‘Guidelines for editing research theses’ prescribed by the Institute for Professional Editors (IPEd).

Closer to home, I thank my precious daughters, Xanthe and Aislin, for whom this research has occupied an inordinate amount of their young lives. It seems the writing of a doctoral thesis is always a long and lonely endeavour, but it took me years to map the cross-disciplinary terrain and hike my way through it, getting lost and found along the way. This research has greatly preoccupied my thoughts, sometimes unhealthily so, making me a distracted father, and for this I forever apologise.

Finally, and most, most importantly, I thank with deep love and gratitude my friend and partner-in-life, Prue. Without her strength and unshrinking support my efforts would have come to nothing.

Abstract i Certificate of Authorship of Thesis iii Dedication v Acknowledgements vii Table of Contents ix Table of Figures xiii Glossary xvii

Chapter 1. Introduction 1 1.1 Initial research question and interdisciplinary context 1.2 Motivation for research 1.3 Statement of problem and framing the research question 1.4 Research aims and objectives 1.5 Premises, claims, and hypotheses 1.5.1 Premises 1.5.2 Claims 1.5.3 Hypotheses 1.6 Approach 1.7 Method 1.8 Delimitations to research 1.9 Chapter outline 1.10 Slipping straitjackets: an unconventional thesis structure 1.11 Definition of key terms

Chapter 2. Film Studies and the Status of Sound 19 2.1 Sound and/or Film Studies? 2.1.1 Introducing the task of theorising cinematic sound 2.1.2 Problems and opportunities for soundtrack scholarship 2.1.3 Sound qua sound (not merely music)

ix | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 2.2 Sound design: history and definition of an ambiguous term 2.2.1 An alternate historiography of sound and cinema 2.2.2 Sound design: history of an ambiguous term 2.2.3 Sound design: definition of an ambiguous term 2.3 Extant literature theorising film-sound 2.3.1 Types of extant film-sound literature 2.3.2 Early sound and cinema (1895–1919) 2.3.3 Classical film-sound theory (1917–1960) 2.3.4 Post-classical film-sound theory (1960–present) 2.4 Framing the gap in Film Studies – beckoning fields of innovation

Chapter 3. Again, What is Cinema? 85 3.1 The status of sound in conventional Film Studies 3.2 Praxis and tacit knowledge 3.3 A working definition of cinematic media 3.4 Multimodal integration and cinematic imagery 3.5 Sound design and embodied meaning 3.6 Perceptual design and narrative design 3.7 Proto-narrative and the cinematic 3.8 Feeling of Body

Chapter 4. Embodied Cognition, Affect, and Meaning 117 4.1 Embodied cognition 4.2 Emotion science 4.2.1 Competing theories of emotion 4.2.2 ‘Primacy of affect’ versus ‘primacy of cognition’ 4.3 Embodied cognition and affect 4.4 Friction at the nexus of scientific and cultural theory 4.5 Summary: embodied cognition, affect, and meaning

Chapter 5. Embodied Simulation and Cinematic Media 133 5.1 Embodied simulation theory 5.2 Function of audition within embodied simulation 5.3 Embodied simulation and cinematic media x | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 5.3.1 Embodied simulation theory and art 5.3.2 Feeling of Body before Theory of Mind 5.4 Cinematic sound and the induction of affect 5.4.1 BRECVEMA framework 5.4.2 Mood-cue approach 5.4.3 Emotional communication theory 5.5 Summary: embodied simulation and cinematic media

Chapter 6. The Body, Consciousness, and Cinematic Media 155 6.1 Consciousness, self, and narrative 6.2 Explicating consciousness 6.2.1 Approaches to theorising consciousness 6.2.2 Types of consciousness – Phenomenal and access consciousness 6.2.3 Types of consciousness – Core and extended consciousness 6.2.4 Types of consciousness – Embodied and narrative self 6.3 Narrative consciousness as animal–culture interface 6.4 Summary: consciousness and cinematic media

Chapter 7. Biological Basis of Cinematic Deep Structure 173 7.1 Cinematic design strategies 7.1.1 Perceptual design and narrative design 7.2 Deep structure of cinematic media 7.2.1 Audiovisual fusion 7.2.2 Audiovisual metaphor 7.2.3 Proto-narrative 7.2.3.1 Definition and function of proto-narrative 7.2.3.2 Proto-narrative as experiential template 7.2.3.3 Music video as proto-narrative 7.2.3.4 Evolution, proto-narrative and narrative 7.3 Body as basis for knowledge 7.4 In the Cut (2003): an embodied simulation approach 7.4.1 Cinematic imagery, image schemas, and conceptual thought 7.4.2 Affective acoustics as embodied spatial schema 7.4.3 Affective zones and cognitive overflow xi | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 7.4.4 Feeling of Body and the construction of character 7.5 Summary: a biological basis for cinematic feeling and thinking

Chapter 8: Conclusions: Embodied Cognition and Cinematic Media 241 8.1 Embodied cognition and the primacy of affect 8.2 Research goals 8.3 Research outcomes 8.4 Research conclusions 8.5 Limitations and caveats to research 8.6 Research contributions and implications 8.7 Directions for future research 8.8 Concluding remarks

Appendix A: Précis of Bartsch & Hübner’s emotional communication theory 253

Appendix B: BRECVEM mechanisms (expanded detail) 255

Appendix C: In the Cut (2003) – selected credits 259

Appendix D: Audiovisual reference materials 261 Supplementary materials: enclosed DVD-ROM

Appendix E: Notes on soundtrack production 263

Appendix F: Curriculum vitae 267

Appendix G: Publications and conference papers 269

Endnotes 271

References 289

Figures 3.1–3.8 92–93 Sound design perceptually freezes moments from a stream of rapidly presented visual information

Figures 3.9 and 3.10 94 Aural beats induce rhythmic entrainment and perceptually freeze affective moments

Figure 3.11 97 Deep visual space but impoverished production of presence

Figure 3.12 97 Enhanced production of presence through multimodal fusion

Figure 3.13 104 Synchronous speech activates the screen sector to which an audience will (mostly) attend

Figure 3.14 105 Shoes scraping on a doormat attracts visual attention from foreground to background action

Figure 5.1 143–144 The original seven BRECVEM mechanisms for the musical induction of emotion

Figure 7.1 175 Perceptual design as the dominant strategy in the Sucker Punch teaser (2011)

Figures 7.2 and 7.3 177 Samurai (left) and Babydoll (right) prepare to fight. Perceptual design orchestrates an aesthetic chills response targeted to the face of Babydoll

Figures 7.5 and 7.6 182 Audiovisual fusion as a basic tool for perceptual design

Figures 7.7 and 7.8 184 Audiovisual metaphor of ‘man-is-machine’

Figures 7.9–7.12 189 The experience of hurtling through space structured by anticipation, apprehension, and peak moments of terror and joy

Figure 7.13 190 Affective structure of The definition of an emotional roller coaster (2009)

Figure 7.14 191 Rollercoaster as proto-narrative. Emotion markers mapped against time and intensity

Figure 7.15 194 Entertainment experiences possess an ‘interest curve’

Figure 7.16 194 Freytag’s Pyramid of dramatic structure

Figure 7.17 196 Interest curves are fractal

Figure 7.18 and 7.19 197 Entrainment and the meshing/unmeshing of audiovisual rhythm to both represent and induce peak aesthetic experience

xiv | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Figures 7.20–7.50 198–200 T-rex battle sequence illustrating the fractal nature of proto-narrative. High density emotional cuing distorts cinematic space and time

Figures 7.51–7.58 202–203 Visual objects are cut-and-pasted and slaved to sonic structure

Figures 7.59–7.64 204 Layered repetition creates a quasi-narrative

Figures 7.65–7.67 212–213 Sound fragments grouped and layered to create large-scale sonic events, emotion cues and markers

Figures 7.68–7.73 214–215 Doors seal off space to create affective acoustic containers

Figure 7.74 215 A door wipes the story to black in the final moments of In the Cut

Figure 7.75 221 Sound design dematerialises the visual image, forewarning danger

Figure 7.76 221 Sound design dematerialises the visual image, forewarning danger

Figures 7.77 and 7.78 223 Returning to New York City at dusk, flickering between bridge girders

Figure 7.79 228 Malloy’s business card swirls in contra-currents of audiovisual space

Figure 7.81 233 Basic embodied source domains and corresponding abstract target domains

1–9 3-stripe a tri-partite soundtrack format. See also DME.

5.1 a form of surround soundfield.

7.1 a form of surround soundfield.

A ADR Automated Dialogue Replacement; a form of post-sync dialogue.

ADSR Attack/Decay/Sustain/Release; a sonic amplitude envelope.

Affect an umbrella term referring to a range of valenced phenomena including (but not limited to) feeling, mood and emotion. Valency is the simple ascription of being either positive or negative, pleasurable or displeasurable, attractive or repulsive, or being in a state of tension or release. Use of the term affect in this thesis follows the terminology used by Juslin & Västfjäll (Juslin & Sloboda 2010, p. 10), and should not be confused with the ‘affect theory’ developed by such theorists as Brian Massumi, Nigel Thrift, or William Connolly. See also: affect, core; emotion; feeling; mood.

Affect, core a primitive, universal and pre-conceptual neuropsychological process which can exist without being labeled, interpreted, or attributed to any cause. It is distinguished by valence and arousal leading to feeling good or bad, energised or enervated. Core affect represents bodily sensations (Lindquist 2013), and is the atom underpinning the psychological construction of emotional experience (Russell 2009). Core affect may be subliminally induced or altered by real or virtual events (such as art, entertainment, imagination). Core affect has special significance for cinematic media because it may be experienced as free-floating mood or causally attributed (albeit erroneously) to target characters, objects, locations and scenarios within the cinematic

xvii| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. environment. Core affect is a concept developed by James A. Russell. See also: virtual reality hypothesis.

ASA Auditory Scene Analysis; a psychophysical model of auditory cognition by which the human auditory system organises sound into perceptually meaningful elements, or auditory streams. The model was developed by Albert Bregman (Bregman 1990, 2008).

AVE Auditory Virtual Environment.

B BI Beat Induction.

BRECVEMA a conceptual framework for the causal explanation of the musical induction of emotion. The BRECVEMA framework consists of eight mechanisms: i) brain stem reflex, ii) rhythmic entrainment, iii) evaluative conditioning, iv) contagion (emotional contagion), v) visual imagery, vi) episodic memory, vii) musical expectancy, and viii) aesthetic judgement. The acronym BRECVEMA is drawn from the names of the mechanisms in phylogenetic order of development. The framework has been developed by Patrik Juslin and Daniel Västfjäll with Simon Liljeström and Lars-Olov Lundqvist (Juslin et al. 2010; Juslin & Västfjäll 2008).

C CGI Computer-Generated Image.

Cinematic imagery affective imagery arising from a mediated spatiotemporal experience. Cinematic qualities are qualities which enhance the affective saliency and presence of the mediated image. Cinematic imagery has the capacity to synaesthetically activate a range of sense modalities. Music and graphic novels, for example, may be said to possess cinematic qualities. Cinematic imagery is a neologism developed during the course of this thesis’ research to underscore the affective qualities inherent in

xviii| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. dynamic multimodal imagery, and resists a simple equation with cinematography. See also: cinematic media; proto-narrative.

Cinematic media media capable of composing, mobilising and transmitting cinematic imagery. Cinematic media in this thesis are recognised, first and foremost, as affective spatiotemporal systems. See also: cinematic imagery.

Cognitive unconscious automatic mental processes which operate beneath the level of conscious awareness to ground an individual in the environment. The cognitive unconscious shapes conscious thought and has been estimated to perform approximately 95 percent of all human thought (Lakoff & Johnson 1999, p. 13). The cognitive unconscious is not to be confused with the Freudian or Jungian conceptualisation of the unconscious.

Consciousness, access (A-consciousness) a state of cognitive manipulation of the content of phenomenal consciousness. Access consciousness is tied to higher cognitive functions such as sequential thought, reasoning and reporting. A-consciousness is a concept developed by Ned Block (Block 1995, pp. 230–236). On Block’s account access and phenomenal consciousness interact but it is difficult to determine the exact nature of interaction because phenomenal experience is largely cognitively impenetrable. See also: consciousness, core; consciousness, extended; consciousness, phenomenal; proto-self.

Consciousness, phenomenal (P-consciousness) a state of phenomenal experience of sensation, feeling, and perception. Phenomenal consciousness is a concept developed by Ned Block (Block 1995, pp. 230–236). On Block’s account phenomenal and access consciousness interact but it is difficult to determine the exact nature of interaction because phenomenal experience is largely cognitively impregnable. See also: consciousness, access; consciousness, core; consciousness, extended; proto-self.

Consciousness, core a state of consciousness produced as the proto-self interacts with the environment. Core consciousness is a transient but continually pulsed representation of the objects and events an organism encounters in the environment formed in conjunction with the organism’s state in interaction with those objects and xix | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. events. Core consciousness generates a unified subjectivity with a unique perspective. It exists within a narrow temporal window with no purview on the past or future moment. Core consciousness is a concept developed by António Damásio (Damásio, AR 1999, pp. 16 and 22). On Damásio’s account core consciousness is scaffolded upon the fundamental layer of the proto-self. Damásio also refers to core consciousness as the ‘core self’. See also: consciousness, access; consciousness, extended; consciousness, phenomenal; proto-self.

Consciousness, extended a state of consciousness composed of a relatively stable but evolving collection of autobiographical facts. It has no access to the current lived moment but situates the core self in both the past and the anticipated future. Extended consciousness is a concept developed by António Damásio (Damásio, AR 1999, pp. 16 and 22). On Damásio’s account extended consciousness emerges from, and is reliant upon, core consciousness. Damásio also refers to extended consciousness as the ‘autobiographical self’. See also: consciousness, access; consciousness, core; consciousness, phenomenal; proto-self.

D DAW Digital Audio Workstation.

Design, narrative a design strategy for the activation and organisation of narrative schema. Narrative design supports psychological immersion into a social environment or storyworld. Narrative design represents and structures emotion and meta-emotion through (mostly) attentional, conscious mechanisms, and is characterised by top-down processing. Narrative design is a term commonly encountered in the computergame industry where it stands in the stead of screenwriting. Use of the term in this thesis signals a desire to differentiate and yet make commensurable the design processes of narrative and perceptual imagery. See also: perceptual design; processing, top-down.

Design, perceptual a design strategy for the activation and organisation of perceptual schema. Perceptual design supports sensory immersion into a virtual environment. Perceptual design inducts core affect, mood and feeling-states within the body of the audience through (mostly) pre-attentional, nonconscious mechanisms, and is xx | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. characterised by bottom-up processing. Use of the term in this thesis signals a desire to differentiate and yet make commensurable the design processes of perceptual and narrative imagery. See also: narrative design; processing, bottom-up.

DME Dialogue-Music-Effects. Acronym referring to a tri-partite soundtrack format comprised of dialogue, music and effects. See also: 3-stripe.

DTS Dedicated To Sound; a series of surround soundfield technologies.

Dolby NR Dolby Noise Reduction.

E EEG Electroencephalography; a form of monitoring electrical brain activity.

Embodiment an existential condition arising from, and constrained by, the sensorimotor capacities of a physical body. Embodiment requires disambiguation because it is a key term within both cultural studies and cognitive science. Within cultural studies, embodiment conventionally refers to meaning that is culturally inscribed in a top-down manner upon the body. Within cognitive science embodiment refers to meaning generated via bottom-up processes as the organism interacts with its environment. Use of the term in this thesis refers to the latter. See also: embodied cognition; embodied meaning; embodied simulation.

Emotion a multicomponent affective process. Emotion is a brief and intense affective state focussed upon a specific object. Use of the term emotion in this thesis follows the terminology used by Juslin & Västfjäll (Juslin & Sloboda 2010, p. 10). See also: affect.

Embodied cognition thesis that the mind is embodied. That is, cognition is shaped by the kinds of experience afforded a body in sensorimotor interaction with the environment within which it is embedded. Embodied cognition is a third-wave research paradigm within cognitive science. Central figures in the development of embodied

xxi | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. cognition are Francisco Varela, Evan Thompson and Eleanor Rosch (Varela, Thompson & Rosch 1991). See also: embodiment; embodied meaning; embodied simulation.

Embodied meaning meaning that emerges bottom-up from the felt qualities of patterns of bodily engagement with the environment. Embodied meaning is afforded, shaped and constrained by the body and its interactions with the world. Mark Johnson asserts all human meaning, abstract thinking and imagination have their basis in our sensorimotor interactions with the world (Johnson, M 2007b, pp. 11–14). See also: embodiment; embodied cognition; embodied simulation.

Embodied metaphor the metaphoric mapping of physical experience onto social phenomena. An example of an embodied metaphor is the conflation of physical proximity with interpersonal warmth such that Being Close = Being Warm. Embodied metaphor correlates physical body states with the affect one feels in that body state. Embodied metaphor and cognitive metaphor are related forms.

Embodied schema Following Mark Johnson (Johnson, M 1987), embodied schema and image schema are interchangeable terms. See also: image schema.

Embodied simulation a mechanism which simulates the experiential knowledge of another’s sensations, emotions and actions. Embodied simulation arises from a bodily format of representation shared with other human beings and supports not only intercorporeality but also intersubjectivity. Embodied simulation is underpinned by the human mirror neuron system (MNS). A ‘liberated’ form of embodied simulation is said to arise from remembered, imaginary or fictional events, and has been proposed to underly the aesthetic and empathic response to art (Gallese 2011c; Wojciehowski & Gallese 2011). Embodied simulation theory rejects the classical ‘sandwich’ model of cognition which privileges cognition as the supreme mediator between perception and action. Embodied simulation theory was developed within the embodied cognition paradigm primarily by Vittorio Gallese (Gallese 2005, 2007a, 2009, 2013). See also: embodiment; embodied cognition; embodied meaning; mirror neuron system (MNS).

Emoacoustics the empirical study of psychoacoustical dimensions of emotional sound design. Emoacoustics is concerned with the mechanisms by which everyday xxii| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. sounds induce affective reactions, and seeks broader explanations than traditional theories of musical emotion. Emoacoustics is the portmanteau of ‘emotional acoustics’. The concept of emoacoustics has been developed by, amongst others, Daniel Västfjäll (Västfjäll, Larsson & Kleiner 2002) (Västfjäll 2003), Ana Tajadura-Jiménez (Tajadura- JiménezLarsson, et al. 2010) (Tajadura-JiménezVäljamäe, et al. 2010), and Aleksander Väljamäe (Väljamäe & Tajadura-Jiménez 2007).

Emotion cues a minimal packet of perceptual information capable of activating the human emotion system within a mediated context. An emotion cue may be delivered via a range of channels; for example, through cinematography or sound design. An emotion cue prompts mood, a necessary precursor to experiencing emotion. Emotion cues may be combined to create larger affective structures such as emotion markers. The term is developed by Greg M. Smith as part of a mood-cue approach to film analysis (Smith, GM 2003b, pp. 29 and 42). See also: affect, core; emotion markers; mood-cue approach.

Emotion marker a cluster of emotion cues orchestrated for the purpose of generating a burst of emotion. Emotion markers serve no narrative function. The term is developed by Greg M. Smith as part of a mood-cue approach to film analysis (Smith, GM 2003b, p. 44). See also: affect, core; emotion cue; mood-cue approach.

Endocept a unit of embodied meaning. An endocept is a low-intensity affective profile associated with an image or concept. An endocept may affectively resonate with, and activate endocepts attached to, other images or concepts. Sundararajan defines an endocept as “an amorphous cognitive-affective structure” (Sundararajan 2000). Endocept is a term developed by Silvano Arieti (Arieti 1976). See also: embodied meaning; proto-narrative.

ERP Event-Related Potential; electrophysical response to a sensory, cognitive, or motor event.

xxiii| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Feeling a subjective experience of an emotion or a mood. Use of the term feeling in this thesis follows the terminology used by Juslin & Västfjäll (Juslin & Sloboda 2010, p. 10). See also: affect.

Feeling of Body (FoB) the experience of a ‘liberated’ form of embodied simulation. FoB is the simulation of the body-states associated with actions, sensations and emotions encountered through a mediated experience such as literature, computergames or cinema. Feeling of Body underpins Theory of Mind by foregrounding the role of the body in sense-making and social interaction. Where Theory of Mind is the simulation of the mind of another (even if fictional), Feeling of Body is the simulation of the feeling state of another (mediated) body. Feeling of Body stands in contrast to Theory of Mind by the former’s capacity to bring forth intercorporeity with the latter’s capacity to bring forth intersubjectivity. The term is developed by Wojciehowski & Gallese (Wojciehowski & Gallese 2011). See also: intersubjectivity; intercorporeity; embodied simulation; theory of mind. fMRI Functional Magnetic Resonance Imaging; a form of neuroimaging.

I Image schema a recurring pattern of sensorimotor experience emerging from interaction with the environment. An image schema structures understanding of embodied experience, and serves as a source domain for metaphorically understanding other experiences. Examples of image schema are Containment, Centre– Periphery, Source–Path–Goal. The concept of an image schema was developed independently by Mark Johnson (Johnson, M 1987) and George Lakoff (Lakoff 1987). Image schema are not mental images. Following Mark Johnson (Johnson, M 1987), image schema and embodied schema are interchangeable terms.

xxiv| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Immersion, narrative a guided form of psychological engagement with a storyworld. Narrative immersion integrates imagery and feelings through the focusing of attention upon story events. Narrative immersion occurs through the simulation of a social environment. Narrative immersion is also referred to as transportation (Green & Donahue 2009) or absorption (Bálint & Tan 2015). See also: immersion, perceptual.

Immersion, perceptual the feeling of being sensorially enveloped in a virtual environment (Biocca & Delaney 1995). Perceptual immersion occurs through the simulation of a physical environment. Perceptual immersion produces (tele-)presence and the feeling of ‘being there’. See also: immersion, narrative; presence.

Infrasound Low-frequency sound below 20Hz. Related to LFE and subsonic.

Intercorporeity a basic, non-conceptual and embodied capacity to sense another’s sensations, emotions and actions. Intercorporeity is a bodily form of empathy. Wojciehowski & Gallese propose intercorporeity to be produced as a consequence of Feeling of Body, and precedes and underpins intersubjectivity. Intercorporeity is equivalent to Merleau-Ponty’s concept of intercorporeality. See also: intersubjectivity; Feeling of Body.

Interoception a composite and usually non-conscious image of the internal physiological state of the body. Interoception provides a basis for the self as a feeling and sentient entity (Craig 2002).

Intersubjectivity a joint sharing of the content of consciousness (such as feelings, thoughts and understandings) between persons. Intersubjectivity appears as a key term, and bears slightly different meanings, across fields such as philosophy, psychology and the social and cognitive sciences.

M MEG Magnetoencephalography; a form of neuroimaging.

Mix-minus DME mix minus dialogue stripe.

MNS: Mirror Neuron System. MN is a class of neuron that fires during both performance of an action and observation of that same action by another. It has been proposed the human MNS is the mirroring mechanism which allows embodied simulation to occur. See also: audiovisual neuron; canonical neuron; embodied simulation.

Mood a prolonged but low intensity affective state which is free-floating because it lacks an object. Use of the term affect in this thesis follows the terminology used by Juslin & Västfjäll (Juslin & Sloboda 2010, p. 10). See also: affect.

Mood-cue approach an approach to film analysis that articulates the ways audiovisual imagery engages the human affective system to induct mood and cue emotions. The approach conceives of a work of cinematic media as an affective ecology where an experience of mood increases the likelihood of experiencing an emotion proper which further sustains (or alters) an experience of mood. The mood-cue approach stands in contrast to the character- and narrative-centric focus of standard film analysis. The mood-cue approach is developed by Greg M. Smith (Smith, GM 1999, 2003a).

N Narrative a representation of temporally and causally linked events. Definitions of narrative, however, are contentious. H. Porter Abbott (Abbott 2008, p. 13), for example, suggests narrative need only a single event but, if comprised of multiple events, requires no causal relationship. Marie-Laure Ryan, acknowledging the lack of an agreed-upon xxvi| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. definition, proposes a ‘fuzzy set’ of narrowing criteria according to spatial, temporal, mental, formal and pragmatic conditions (Ryan 2007, pp. 28–31). Ryan also observes a robust definition should support assumptions that narrative is concerned with conflict and problem-solving, interpersonal relations, and the temporal nature of human experience (Ryan 2007, p. 24). This thesis assumes narrative seeks to represent and explain the behaviour and mental life of real or fictional characters, and may therefore be considered a form of social cognition. Narrative is considered within cognitive psychology to be the organising principal for conscious thought and the construction of a Theory of Mind (ToM). See also: Theory of Mind.

P PECMA flow model a general theory of film experience. PECMA (acronym for Perception, Emotion, Cognition, Motor Action) refers to the flow of information from perception through affective and cognitive processing to motor action. PECMA flow is described in a strongly bottom-up manner (from perception to action) although flow may be reversed via top-down processes based upon personal or cultural learning. The PECMA flow model has been used to explain a variety of aesthetic phenomena, and the differences between narrative and associative forms. PECMA flow in cinematic experience is proposed to occur in ways similar to real life (including dreams and planning for the future) because it originates and runs on the same embodied mind architecture. Unlike in real life or interactive media, such as VR and computergames, the motor aspect of the PECMA flow within cinema is experienced only as simulation. The PECMA flow model is an evolutionary biocultural approach to understanding cinema, and is developed by Torben Grodal (Grodal 2006, 2009b).

Perceptual symbol a percept, or “a record of the neural activation that arises during perception” (Barsalou 1999, p. 583). Patterns of sensorimotor activation occurring during perception are captured and transformed into a neural code resulting in a perceptual symbol. During cognition, perceptual symbols are used to partially reactivate the sensorimotor qualities they represent. The storage and reactivation of perceptual symbols operates not at the level of holistic experiences but at the level of perceptual xxvii| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. components (such as texture, colour, heat, sound, etc.). Over time, these components become associated and organised in memory around common frames to implement a basic conceptual system. The theory of perceptual symbol systems was developed by Lawrence W. Barsalou (Barsalou 1999). Perceptual symbols are also termed cogs by Gallese & Lakoff (Gallese & Lakoff 2005). See also: cognition, embodied; image schema.

PET Positron Emission Tomography; a form of neuroimaging.

POA Point-of-audition.

POV Point-of-view.

Presence the perceptual illusion of non-mediation (Lombard & Ditton 1997). The experience of presence can manifest in physical, social, or self-presence forms. The feeling of presence is the feeling of ‘being there’ which sits at the heart of all mediated experiences, from literature to VR technologies. Use of the term presence in this thesis refers to tele-presence, also known as virtual presence or mediated presence. See also: immersion, perceptual; immersion, narrative.

Processing, bottom-up an innate process by which perception results from sensory information from the environment being processed ‘upwards’ in an emergent fashion to produce an ‘image’. Bottom-up processing is stimulus-driven. See also: processing, top-down.

Processing, top-down a process by which perception results from the internal models derived from prior knowledge applied ‘downwards’ to structure incoming sensory information from the environment. Top-down processing is theory-driven. See also: processing, bottom-up.

Proto-narrative a minimum unit of meaning which functions below the threshold of narrative. Proto-narrative is bound together by an associative affective logic devoid of temporal causal connections (Sundararajan 2008). Affective logic of this kind affords high malleability and connectability capable of generating networks of embodied xxviii| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. meaning. Proto-narrative is non-linear and pre-linguistic yet underpins narrative proper. Proto-narrative resembles Silvano Arieti’s concept of ‘endocepts’ (Arieti 1976) and what Mark Turner calls ‘small stories’ (Turner 1996). See also: affect, core; embodied meaning; endocept.

Proto-self a primitive nonconsciousness consisting of continually updated neural mappings of the physical state of an organism. The proto-self is the fundamental layer of reference upon which core and extended consciousness are scaffolded. The proto-self is a concept developed by António Damásio (Damásio, AR 1999, p. 153). See also: consciousness, access; consciousness, core; consciousness, extended; consciousness, phenomenal.

S SDDS Sony Dynamic Digital Sound; a form of surround soundfield.

S/N ratio Signal-to-noise ratio.

Somatic marker hypothesis the neuroscientific hypothesis that human decision-making and behaviour is guided by ‘somatic markers’ arising from affective and bioregulatory processes. The somatic marker hypothesis is developed by António Damásio, Daniel Tranel, and Hanna Damásio (Damásio, AR, Tranel & Damásio 1991).

T THD Total Harmonic Distortion.

Theory of Mind (ToM) the ability to attribute mental states to oneself and to others. This capacity affords explanation, prediction and coordination of human behaviour. The representation of another’s mental state is also referred to as mind- reading. Theory-Theory and Simulation-Theory are two competing explanatory xxix| ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. approaches to the phenomenon of mind-reading. Theory-Theory, as the name suggests, proposes another’s mind may be understood via a theory based upon observable external states, such as behaviour. Simulation Theory, however, proposes the mind of another may be simulated through the re-use of one’s own mental processes and resources. A central figure in the development of Simulation Theory is Alvin Goldman (Goldman 1992). The position adopted in this thesis considers Theory of mind (ToM) to be mental simulation. Theory of mind affords the capacity for intersubectivity. Theory of mind (ToM) stands in contrast to Feeling of Body (FoB). See also: intersubectivity; Feeling of Body.

V VE: Virtual Environment.

Virtual reality hypothesis an hypothesis proposing an encounter with virtual events and environments (as art, entertainment, or imagination) produces core affect in the same way as encountering real events and environments. (Core affect is the fundamental atom underpinning the psychological construction of emotional experience.) The adaptive purpose is the capacity to experience and understand how one might feel in response to possible events and actions, such as in flights of imagination or planning future events. The virtual reality hypothesis was developed by James A. Russell (Russell 2003, 2009). See also: affect, core.

VR: Virtual Reality.

VFX: Visual effects. See also: CGI.

W X Y Z

What is important now is to recover our senses. We must learn to see more, to hear more, to feel more.

Our task is not to find the maximum amount of content in a work of art, much less to squeeze more content out of the work than is already there. Our task is to cut back content so that we can see the thing at all … In place of a hermeneutics we need an erotics of art. – Susan Sontag, ‘Against interpretation’ (1966, p. 14, emphasis in original)

1.1 Initial research question and interdisciplinary context What is the nature and function of sound design in cinematic media? This thesis departs from standard notions of cinematic sound by examining its meaning-making functions from the perspective of embodied cognition, a research programme within contemporary cognitive science1. A framework of this kind develops an account of cinematic media which puts into play Mark Johnson’s concept of ‘embodied meaning’ which asserts that all human meaning, abstract thinking and imagination have their basis in our sensorimotor interactions with the world (Johnson, M 2007b, pp. 11-14). Rejecting the notion that meaning belongs solely to the domain of language, Johnson observes that “[m]eaning traffics in patterns, images, qualities, feelings, and eventually concepts and propositions” (2007b, p. 9). As this thesis will show, an embodied cognition approach to cinematic media reveals the primary function of sound design to be the elicitation of affect which, in turn, shapes and guides the cognition and consciousness of the individual experiencing the mediated imagery.

The paradigm of embodied cognition and, in particular, the theory of embodied simulation (Gallese 2005) deriving from it (aspects of which I will deploy throughout this thesis) represent a divergence from mainstream theorising of cinema, and offers Film Studies opportunities for a broader and more flexible understanding of its object of study. My research seeks to expand our knowledge of the nature and function of sound design in cinematic media by adding fresh insights to this emerging new field.

Although embodiment has become an influential concept within Film Studies2, embodied simulation theory has only recently been explicitly applied to cinema (Gallese & Guerra 2012b, 2012a). Preceding this, however, is the work of Fahlenbrach. Drawing upon concepts from cognitive linguistics, Fahlenbrach has written extensively upon cinematic sound and affect from the perspective of embodied metaphor (Fahlenbrach 2002, 2005b, 2005a) and, latterly, by developing what might be considered an embodied simulation approach (Fahlenbrach 2007, 2008). My own research (Ward, MS 2010, 2015; Ward, MS & Leung 2008), which interrogates my implicit knowledge drawn from professional praxis, intersects with and extends the research of Fahlenbrach. Specifically, my contribution rests in formalising an embodied simulation account of cinematic sound, and locates a basis for embodied meaning in the sonic induction of affect through the application of the BRECVEMA framework (Juslin 2013b; Juslin, Barradas & Eerola 2015; Juslin, Harmat & Eerola 2013; Juslin et al. 2011; Juslin et al. 2010; Juslin & Västfjäll 2008). Furthermore, I add sophistication to this framework by combining it with an enhanced understanding of the production of a ‘Feeling of Body’ (Wojciehowski & Gallese 2011).

Historically, Film Studies has tended to emphasise the visual while dwelling upon the narrative by asking questions of cinema which may only be answered by vision or narrative. Such visual narrativism has proceeded to the detriment of productive sound theory. An embodied cognition approach offers a common denominator across all aspects of cinematic media, narrative and non-narrative alike. An embodied cognition approach, aspects of which I will utilise in this thesis, puts Film Studies in contact with evolutionary and neurobiological theories of emotion and cognition, and in doing so manifests an “ecological metatheory” of the kind called for by Joseph and Barbara Fisher Anderson (1996).

Within the field of Film Studies, and not withstanding the recent surge in sound-centric publications, sound design remains under-theorised. However, it is precisely because of its marginalised status that sound offers an opportunity by which to investigate afresh the underlying principles of cinematic media. My thesis is not especially attentive to the usual targets of speech and music in soundtrack discourse. Instead, it focuses upon that difficult-to- classify and mongrel form of sound often referred to as noise. Noise is comprised of sound effects, Foley and atmospheres. Noise is environmental sound, everyday sound. Sound of this kind is the most under-theorised element of the soundtrack. As a consequence, the design of

2 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. environmental sound offers exceptionally fertile yet under-examined space within which fresh theories of cinematic media may be modelled. Specifically, my thesis argues for the primacy of affect within the meaning-making architecture of cinematic media in general and sound design in particular 3. This architecture is comprised of the biologically-driven processes of embodied simulation (Gallese 2005) and its corollary of ‘Feeling of Body’ (Wojciehowski & Gallese 2011). These claims are distilled from my 30 years of professional praxis, and are substantiated through contemporary theories in cognitive science.

1.2 Motivation for research My own accrued tacit knowledge about the workings of sound in cinematic media deeply motivates this research. Over the course of my professional life as a sound designer on a range of feature film, documentary and television projects, I have noticed that my colleagues and I regularly employ strategies which cannot be fully described nor explained by the standard Film Studies paradigm of visual narrativism. This mismatch between the tacit knowledge of professional sound design praxis and the explicit knowledge of theoretical orthodoxy reveals a gap which requires explanation and reconciliation, and serves to motivate my research. What does cinematic sound design actually design, and to what purpose?

1.3 Statement of problem and framing the research question The standard account of film-sound4 within classical and contemporary Film Studies holds the function of sound to be the support of visual imagery in the support of narrative. This is the paradigm of visual narrativism. The underlying mechanisms by which sound achieves these functions, however, have not been satisfactorily captured within Film Studies while other phenomena uniquely associated with sound have been elided. This deficiency, detailed in Chapter 2, springs from an assumed hierarchy of significance which privileges visuality and narrativity. As a consequence, the standard account of film-sound fails to fully articulate the function of sound design within cinematic media and leads to the question, paraphrasing Brophy (2008), “Where is sound in film theory?”5

During its embryonic years, Film Studies routinely imported visual and narrative theory to explain its object. The hegemonic effect of the interlocking logics of visuality and narrativity formed questions to be answered through the study of vision or narrative while inhibiting

3 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. questions better answered by other modes of inquiry. Visual narrativism has roots deep in the Western philosophical tradition reaching through Plato at least as far back as Heraclitus and Parmenides (Levin 1993a). As the literature review in Chapter 2 reveals, the philosophical and cultural legacy of hegemonic vision underpinning the classical model within Film Studies offers only a weak account of cinematic sound. In response, I develop an alternative account using an embodied cognition approach. This account abandons the foundational visual narrative assumptions of classical film theory by arguing that the essence of cinematic imagery is not merely visual but multimodal, and not narrative but affective. Thus, my embodied cognition approach places embodied affect, presence and immersion at the centre of the inquiry.

1.4 Research aims and objectives The theoretical objectives of my research include the identification of historical and contemporary concepts and definitions of cinematic sound design, and the identification of current cinematic sound design practices which license a revision of these concepts and definitions.

The theoretical aim of my research is to explain the function of sound design within cinematic media by developing a broad and inclusive framework which takes an embodied and affective approach, and which situates sound design as an active participant within the cinematic experience.

The practical aim of my research is to establish a set of criteria and conceptual tools by which sound design may be modelled and critiqued.

The methodological aim is to explore cinematic sound through an interdisciplinary approach which is based in the humanities but draws upon concepts arising from the perceptual, affective and cognitive sciences. The methodology is also informed by the experiential knowledge of my professional praxis6.

4 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 1.5 Premises, claims, and hypotheses In the introduction above, a number of ideas and propositions were lightly touched upon. By way of providing a map of the intellectual terrain we are about to traverse, I present below a detailed series of premises, claims and hypotheses which give a more nuanced structure to that terrain, and which indicate the trajectory to be taken by this thesis. I will flesh out these premises, claims, and hypotheses throughout the full extent of my thesis.

1.5.1 Premises Premise 1: Film Studies is shaped by the legacy of Platonic philosophy, and is characterised by a concern with vision as the primary carrier of knowledge7.

Premise 2: Film Studies is shaped by a focalisation upon visuality and narrativity.

Premise 3: Film Studies has failed to explain the function of sound design in a broad and inclusive way.

Premise 4: The functions of sound design are integral to the functions of cinematic media.

Premise 5: The rules governing human perception, emotion, and cognition emerged from the pressures of evolutionary biology. These rules are analogous across real and mediated worlds. Cinematic media, therefore, emerged in ecological relation to human perceptual, affective, and cognitive biases.8

Premise 6: The functions and principles of cinematic sound design may be discovered through the close exploration and analysis of creative praxis.

1.5.2 Claims Claim 1: Cinematic media are not merely visual but multimodal in nature.

Claim 2: Cinematic media are primarily affective in nature.

Claim 3: The primary function of cinematic sound design is the production of affective content through embodied processes.

5 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Claim 4: Cinematic sound design elicits core affect9 which in turn gives rise to feelings, moods and emotions.

Claim 5: Cinematic sound design mobilises the social and cultural processing of core affect, feelings, moods and emotions to produce yet other classes of affect, such as meta-emotions10.

Claim 6: Cinematic sound design elicits and represents feelings, moods and emotions, and such audiovisual affect guides and confers the meaning of places, things, people and events.

Claim 7: Audiovisual emotions and meta-emotions guide the construction and comprehension of narrative in cinematic media.

Claim 8: Cinematic sound design is an invitation to ‘play at being’ and to feel as another feels11.

1.5.3 Hypotheses Hypothesis 1: The primary function of cinematic sound design is the production of embodied meaning. Explicitly, the design of cinematic sound performs a meaning-making function by guiding the activation of embodied processes which create affective content.

Hypothesis 2: Cinematic sound produces affective content via the interaction of two design strategies: i) perceptual immersion, and ii) narrative immersion. Perceptual immersion elicits core affect, mood and feeling-states within the body of the audience through (mostly) ‘bottom-up’ nonconscious12 mechanisms while narrative immersion activates representational structures of emotion and meta-emotion through (mostly) ‘top-down’ conscious mechanisms13.

Hypothesis 3: Cinematic sound design gives rise to ‘Feeling of Body’ (FoB) and intercorporeity which, in turn, supports the emergence of ‘Theory of Mind’ (ToM) and intersubjectivity, and thus invites empathy with a mediated or fictional other’s affective state.

Hypothesis 4: An embodied cognition approach – specifically, an embodied simulation account14 – provides a unique, broad, flexible and robust framework for modelling the purpose and function of cinematic sound.

Hypothesis 5: An embodied cognition approach to cinematic sound design provides a model fundamental to all cinematic media.

1.6 Approach Precisely because the nature of theorising cinematic sound sits stubbornly at the crossroads of art and entertainment, emotion and cognition, and aesthetics and the cognitive sciences, an interdisciplinary approach is required. Through such an approach I am able to leverage insights from the ‘third wave’ of cognitive science into a humanities workspace.

Interdisciplinarity of this kind generates new fields of research, most notably neuroaesthetics15. Neuroaesthetics, which received a formal definition only as recently as 2002, is the “scientific study of the neural bases for the contemplation and creation of a work of art” (Nalbantian 2008, p. 357). One of the hallmarks of neuroaesthetics is its capacity to facilitate dialogue between the ‘Two Cultures’ of the sciences and humanities, particularly those disciplines which were considered antithetical or lacked a common theoretical frame through which to engage. Significant figures in establishing the field of neuroaesthetics are Semi Zeki (1999a, 1999b, 2009) and Vilayanur S. Ramachandran (1999), both neuroscientists working at the cusp between biology and psychology16. Arthur Shimamura (2011, 2013a), in particular, has strongly laid out the foundational concepts and issues for an empirical aesthetics of art and cinema. I will be drawing on ideas from this interdisciplinary field as I develop my own thesis about sound and cinematic media.

Admittedly, for someone such as myself operating within the humanities and cultural industries, there exists a serious risk of misappropriating scientific concepts without proper regard to their context or provenance. The dangers of over-reaching when working in such an interdisciplinary way are very real17. In spite of these risks, I most decidedly use an interdisciplinary approach to draw out and clarify the conceptual relationships I believe to exist between cinematic media, aesthetics, and the cognitive sciences. This strategy is intended to dissolve the conceptual impasse in classical film theory, which lingers still in Film Studies, and which acts to disqualify sound as an active participant in the meaning-making processes of contemporary media. In this regard I follow Patrick Colm Hogan when he says,

7 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. … I believe that [Mark] Turner is correct that humanists must at least be among the leaders in the cognitive revolution. Literature and the arts pose specific problems for cognitive study; they raise specific issues; they present specific challenges. A neurobiologist who turns briefly to literature as a side issue is unlikely to do it justice. Of course, humanists have the converse problem. They need to achieve familiarity with cognitive methods and principles. (Hogan 2003, page 3)

Achieving familiarity with cognitive methods and principles, of course, is no simple task. To counter this deficiency, my method, detailed below, is grounded in what I already know as a consequence of my own professional creative praxis, albeit tacitly in the first instance. In showing how cinematic sound design produces affective content (and how this affective content may bring forth meaning, intercorporeity, intersubjectivity and empathy), I integrate four distinct theories to create a conceptual framework. Each theory focuses upon the elicitation or representation of affect at discrete functional levels of description. For ease, I refer to this conceptual framework as an embodied simulation account of cinematic sound design in recognition of the capacity of Gallese’s embodied simulation theory (2005) to contain, integrate and structure all levels of description. The four-tiered conceptual framework, from elemental to compound, is comprised of: i) the BRECVEMA framework (Juslin 2013b; Juslin, Barradas & Eerola 2015; Juslin, Harmat & Eerola 2013; Juslin et al. 2011; Juslin et al. 2010; Juslin & Västfjäll 2008) ii) the Mood-cue approach (Smith, GM 1999, 2003a) iii) Emotional communication theory (Bartsch 2004, 2005, 2010; Bartsch & Hübner 2005; Bartsch & Oliver 2011); and lastly, iv) Embodied simulation theory (Freedberg & Gallese 2007; Gallese 2005, 2007a, 2009, 2011a, 2013; Gallese & Sinigaglia 2011a).

I will show how the explanatory strength of this four-tiered conceptual framework is its highly granular and scalable capacity to describe how cinematic sound may produce embodied meaning. From micro to macro, the first tier is the BRECVEMA framework which describes the auditory induction of core affect and emotion. The second tier is the mood-cue

8 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. approach which describes the audiovisual induction of filmic emotion18. The third tier is emotional communication theory which describes the tertiary level of social and cultural processing in the production of ‘meta-emotions’. Lastly, the fourth tier is embodied simulation theory which serves to integrate the preceding three tiers to explain the emergence of intercorporeity, intersubjectivity, and empathy.

A key feature of this conceptual framework is the recognition of emotion and cognition as entwined embodied processes. It explains how audition may induce or represent an affective experience, how that affective experience may frame attention, cognition and consciousness, and how affect provides the basis for intra- and interpersonal communication. Affective content, therefore, can be said to be the lingua franca between each tier of the framework. These theories are complementary and scaffold to outline an embodied, enactive and affective approach to cinematic sound design.

1.7 Method The embodied cognition approach of my thesis is interdisciplinary in nature and proceeds from the bedrock of creative praxis19. It maps insights gained from my own creative practice against cognate concepts emerging from the cognitive sciences, and these concepts are subsequently re-mapped into a broader humanities context. This method recognises art and science as complementary modes of knowing and, in this way, intends to counter the deficiency identified above by Hogan (2003, page 3).

The interdisciplinary nature of this method bears repeating. In referencing C.P. Snow’s famous notion of the ‘Two Cultures’ (1956) which identifies a schism between the sciences and humanities, Amy Ione speaks of the historical bifurcation of tacit and explicit modes of knowledge when describing the artist performing a delicate balance between praxis and theory:

[E]nthusiasm for the complex conundrum of consciousness has offered a means to revisit all ways of knowing and, as I have argued elsewhere (Ione 2000, 2002, 2005), the science/humanities pairing has a history that relegated art to a secondary position. For example, we can date the elevation of language over visual art back to Plato’s disdain for representation. It is also well known that with the early Renaissance, artists

9 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. further complicated our ability to ‘see’ art on its own terms by highlighting rhetoric and theory in a desire to elevate their professional position. By downplaying praxis and experimentation, the artists succeeded. The effect of their success, however, remains with us today in a way that is particularly evident when we place art within the narrow ‘two cultures’ matrix that pits science against the humanities. (Ione 2006, p. 28, emphasis in original)

For Ione, the partitioning of practice from theory, art from technology, and humanities from science is unacceptably restrictive, but the artist is well situated to confront this schism and make whole all human knowledge, or, to borrow E.O. Wilson’s term, perform an act of ‘consilience’ (1998). Patrick Colm Hogan (2003) and Edward Slingerland (2008a; 2011b), both humanities professors, echo Wilson in advancing a consilience which takes seriously a re-integration of the sciences and humanities20. My thesis has been composed in this spirit.

1.8 Delimitations to research A certain kind of risk arises when taking up an embodied cognition approach to cinematic media. One’s argument risks being destabilised – if not overwhelmed – by the kaleidoscopic developments in contemporary theories of embodied cognition. Firm delimitations to the research are therefore required to manage an effective scope.

What is this research concerned with, and what is it not? This thesis is concerned with cinematic sound design21. It is not necessarily concerned with music or voice per se unless music or voice illuminate an understanding of environmental sound, and only in so far as it provides insights into what William Gaver calls ‘everyday listening’ (1986, 1993a, 1993b). Everyday listening developed under evolutionary pressures, and these pressures selected for the extraction of meaningful environmental information. Environmental sound is under- theorised within the literature, but its significance becomes clarified when placed into an evolutionary context which recognises “speech and music originated from a common emotional signal system based on the imitation and modification of environmental sounds” (Ma & Thompson 2015, p. 14563). Plainly put, the laws which govern the cognition of speech and music are parasitic upon our body’s evolved biological and affective responses to environmental sound.

10 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Furthermore, in recognising the cinematic experience as a multimodal event, the scope of this research focuses upon moving pictures only in so far as investigating the crossmodal interaction which occurs between aural and visual imagery. The intention is to counterbalance the ocularcentric tendencies of conventional film theory by acknowledging the complex contributions both audition and vision make to the cinematic experience because to dwell upon the aural to the exclusion of the visual is to reproduce in reverse the very same problems of ocularcentrism.

1.9 Chapter outline The investigation of cinematic media through the filter of embodied cognition theory is bedevilled by many varied but interwoven conceptions of imagery, perception, emotion, and cognition. How may such relationships be grasped when theories of embodied cognition have a tendency to blur traditional conceptual boundaries? To capture such complexity this thesis is marshalled into eight chapters.

Chapter 1: Introduction Chapter 1 introduces the problem of theorising sound design within conventional Film Studies. It announces the author’s experience as a professional sound designer, identifies a mismatch between theory and praxis, and leverages this mismatch to frame the research question. Research aims and objectives are identified as well as premises, claims and hypotheses. Several delimitations are described, and key terms are identified.

Chapter 2: Film Studies and the Status of Sound Chapter 2 presents the state of sound theory within the field of Film Studies. Functioning in part as a contextual literature review, the chapter identifies four thematic preoccupations: visualism, narrativism, rationalism, and the belief that meaning is solely determined by language and culture. These preoccupations act as conceptual straitjackets and impoverish the status of sound. The review problematises the assumed primacy of narrative by reframing the tension between spectacle and narrative. The review reveals the limited utility of standard film theory to explain the practice, effects, and meaning of cinematic sound design, and frames this gap in knowledge as a research question requiring an interdisciplinary approach.

11 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Chapter 3: Again, What is Cinema? Chapter 3 takes up where the literature review leaves off by asking that most Bazinian question: What is cinema? It does this from the perspective of professional practice. It focuses upon the tacit knowledge which underpins the praxis of sound design, and makes claims about the multimodal and affective primacy of cinematic media by exploring the relationship between multimodality, cinematic imagery, mental imagery, and consciousness22. The chapter develops a working definition of the cinematic, and introduces the terms perceptual design and narrative design, proto-narrative, and Feeling of Body. In particular, perceptual design and proto-narrative are used to describe how nonconscious affective processes are activated by cinematic media. At first blush, these professional intuitions appear both contentious and counterintuitive, and so Chapter 4 recruits insights from the cognitive sciences to substantiate these claims.

Chapter 4. Embodied Cognition, Affect, and Meaning Chapter 4 adopts an interdisciplinary stance to provide important information for grounding the reader in the embodied cognition research programme within the cognitive sciences. It examines several key concepts including the primacy of affect (Zajonc 1984) and embodied meaning (Johnson, M 2007b, pp. 10-11). The chapter also contextualises the concept of embodied emotion (Niedenthal 2007; Niedenthal & Maringer 2009; Winkielman, Niedenthal & Oberman 2008; Winkielman, Niedenthal & Oberman 2009) by placing it within the functional architecture of affect, art and entertainment developed within contemporary theories of media psychology (Tan, ESH 1994b, 1999, 2000, 2008; Visch, Tan & Molenaar 2010; Vorderer 2001; Vorderer & Hartmann 2009; Vorderer, Klimmt & Ritterfeld 2004).

Chapter 5. Embodied Simulation and Cinematic Media Chapter 5 specifies the embodied simulation (ES) account of cinematic sound. It uses Vittorio Gallese’s theory of embodied simulation (Freedberg & Gallese 2007; Gallese 2005, 2007a, 2009, 2011a, 2013; Gallese & Sinigaglia 2011a) as a unifying theoretical framework for the kinds of experiences provided by cinematic media. Central to the account is the function of audition and affect within embodied simulation, and the application of these insights to the general function of cinematic media. Specifically, a novel framework is proposed which nests theories for the induction and structuring of affective auditory content: i) the BRECVEMA framework developed by Patrik Juslin, Daniel Västfjäll and colleagues (2013b; 2015; 2013; 2011; 2010; 2008); ii) the mood-cue approach developed by Greg M. Smith (1999, 2003a);

12 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. and iii) emotional communication theory proposed by Anne Bartsch and Susanne Hübner (2004, 2005, 2010; 2005; 2011). Furthermore, the concepts of embodied meaning, Feeling of Body, and liberated embodied simulation are used to explore cinematic media’s capacity to generate intercorporeity, intersubjectivity, and empathy.

Chapter 6. The Body, Consciousness, and Cinematic Media Chapter 6 explores the capacity of cinematic media to emulate the embodied metaphoric basis of imagination and narrative consciousness. It develops a parallel between perceptual immersion, phenomenal consciousness and presence on the one hand and narrative immersion, access consciousness and the autobiographical self on the other. From an embodied simulation perspective, the chapter explores the putative schism between body and mind, and its cinematic doppelgänger of spectacle and narrative. Superficially, an ES account appears to promote a form of mind–body dualism; however, the human mind is profoundly embodied because conscious thought is rooted in, and a virtualisation of, the body’s sensorimotor understanding of the physical world. The human social world of conscious thought, self, and culture is scaffolded upon this embodied knowledge.

Chapter 7. Biological Basis of Cinematic Deep Structure Chapter 7 is the largest and perhaps most substantial chapter in the thesis. In many ways it is the climax of the argument. The chapter explores the aesthetics and embodied basis of cinematic deep structure. Cinematic media, through the technological recombination of aural and visual imagery, routinely engineers a form of synaesthetic experience. When manipulated, the body’s innate processes for multimodal fusion produce novel perceptual symbols which, in turn, stimulate multimodal metaphor and cognitive blending. At the heart of these embodied processes are affective sensations of movement, and, when organised into patterns, form the basis of proto-narrative. Proto-narrative acts as a template upon which narrative proper is based. The chapter uses embodied simulation theory to analyse the aesthetics and design strategies of cinematic imagery. Illustrations of these strategies are taken from my own work and the work of other sound designers. Specifically, I examine my sound design strategies used in Jane Campion’s In the Cut (2003). I also include a range of examples drawn from music video, television commercials, and arthouse and blockbuster movies.

13 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Chapter 8. Conclusions: Embodied Cognition and its Implications for Cinematic Media Chapter 8 summarises from an embodied cognition perspective how cinematic media elicits and represents affect, creates presence and immersion, shapes attention, and frames cognition. It explains cinematic media to be a technological homologue to an embodied and narrative consciousness. Finally, I consider how my findings are important to the understanding and production of contemporary moving image media, and how these findings can better inform decisions made by individuals and institutions involved in policy formation, media education and cultural production now and in the future.

1.10 Slipping straitjackets: justifications for unconventional thesis structure Before proceeding to these chapters to argue for the value of an embodied simulation account of cinematic media, I pause to explain why sound needs to be slipped from the four conceptual straitjackets that will be identified in the literature review. In offering this explanation in the next few paragraphs, I am keen to help the reader understand, from the outset, why the thesis proceeds as it does.

First, the four conceptual straitjackets. Number one is the assumption that vision exercises hegemony over audition. The second is the assumption that cognition commands dominance over emotion. The third is the belief that meaning is the dominion of language and culture. The fourth is the assumption that consciousness and narrative command significance over the body’s sensorimotor system and its associated imagery23. Not one of these is insignificant, and all four biases are endemic in Film Studies. Collectively, they pinion cinematic sound and present obstacles to theories of embodied cognition. Therefore each of these precepts must be addressed before one can argue for the validity and significance of an embodied simulation approach to cinematic media.

Hence the thesis structure. Because of the entangled nature of these straitjacketing precepts, my thesis exhibits a somewhat unconventional structure. Chapter 2 fulfils the usual role of a literature review but it does not entirely suffice. The thesis draws upon film practice, film theory and cognitive science, and I cannot assume a typical humanities scholar to be versed in cognitive science, particularly theories of embodied cognition. Nor may I assume a typical cognitive scientist to be familiar with considerations of media aesthetics and dramaturgy. Nor may I assume either of these readers to command the first-hand experience of a mediamaker

14 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. in the design and crafting of cinematic media. The task of this thesis, however, is to communicate to all three audiences across these varied domains in a cohesive and coherent way. As a consequence, I must give the reader all the necessary concepts from each of these domains in coming to grips with my argument for an embodied simulation approach to cinematic media.

It is for these reasons the literature review provided in Chapter 2 must be augmented. Specifically, Chapter 3 reviews and critiques some extant literature as it dispels the myth of visual sovereignty and ocularcentrism by, firstly, describing the profoundly multimodal nature of human beings and, secondly, by explaining the consequences of this multimodality for media theory. Chapter 4 unstitches the assumption that cognition eclipses emotion by arguing that cognition is rooted in the human affective system, and, through an examination of Mark Johnson’s theory of embodied meaning (2007b, pp. 10-11), I challenge the conventional understanding of meaning as linguistically and culturally determined. Chapter 6 displaces narrativism by critiquing the assumptions of mind–body dualism which privilege consciousness over the human sensorimotor system. Each of these arguments draws from a burden of evidence accumulating in contemporary cognitive science, and have vast consequences for the philosophical positions historically entrenched within the arts and humanities. I therefore have to spend time in each of these chapters addressing some specific terms, themes and caches of literature that were not able to be encompassed in the literature review chapter. This strategy of augmentative literature review is intended to diminish the problem of discipline-specific terminology and definitions which cripple communication between artists and scholars of the humanities and sciences.

1.11 Definition of key terms To end this introduction and to constrain and clarify the multitude of assumptions within each of the disciplines to which this thesis connects, here at the outset I offer my working definitions of key terms and concepts. These definitions, more of which are to be found in the glossary at the front of this thesis, lead us into the chapters that I have outlined in this Introduction.

15 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Sound design Sound design is a general creative process by which sound fragments are created, selected, organised, and blended into perceptually unified and coherent ‘sound objects’. These sound objects form sound events and are the basis for auditory imagery24. Sound design conventionally refers to the creation of those elements which are neither speech nor music – namely, sound effects, Foley and atmospheres – and it is these elements which are the central focus of this thesis25. From a techno-historical perspective, sound design exhibits a tendency towards developing practices and technologies with greater and greater immersive characteristics. However, sound design remains an ambiguous and contested term, and is detailed further in Chapter 2.

Cinematic media Cinematic media is a neologism developed during the course of my research to reflect the idea that cinematic imagery is not merely moving pictures and not only located in cinema. It acknowledges moving imagery may possess, to a greater or lesser degree, cinematic qualities. Cinematic qualities of the image are affective qualities which produce an enhanced saliency and presence. Perhaps counterintuitively, cinematic qualities are not limited to film per se but may be found across a diverse range of media26. Cinematic media, therefore, are those media capable of transmitting affective moving imagery. Cinematic media is described in further detail in Chapter 3.

Affect Affect is a key term within both the sciences and humanities. However, the sciences and humanities conceptualise affect very differently and, as a consequence, have produced definitions which are largely incompatible. To make matters more confusing, affect is often erroneously used interchangeably with emotion. For the sake of clarity and consistency, my definition follows that used by Juslin & Sloboda (2010, pp. 9-10); that is, affect is an ‘umbrella term’ which refers to a range of different phenomena such as core affect, feeling, mood and emotion. My reference to theories of affect should not be confused with ‘affect theory’ currently in use within humanities discourse because my use of the term is not identical to its use by affect theorists such as Brian Massumi (2002a), Nigel Thrift (2004; 2008), or William Connolly (2002). Affect is discussed in further detail in Chapter 4.

16 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Embodiment Embodiment is a key term in both cultural studies and cognitive science but is conceptualised in radically different ways. Within cultural studies, embodiment conventionally refers to the manner in which cultural meaning is ‘inscribed’ in a top-down manner upon the body. That is, identity and cultural value are ‘embodied’. Within cognitive science, embodiment refers to bottom-up biological processes – usually innate, automatic and nonconscious – which generate meaning for the organism as it interacts with its environment. That is, the mind is ‘embodied’. My use of the term is the latter. In this way, I follow Gallese & Sinigaglia when they state embodiment “usually means that body parts, bodily actions, or body representations play a crucial role in cognition” (2011a, p. 513). According to Niedenthal & Maringer, referencing Barsalou, “mental processes are called ‘embodied’ because an incomplete but cognitively productive reexperience is produced in the brain as if the individual were there in the very situation, the very emotional state, or with the very object of thought” (2009, p. 122, emphasis in original). Rather than meaning becoming ‘inscribed’ upon the body as if it were a blank slate, my use of the term assumes the body gives rise to meaning.

Embodied simulation Embodied simulation is a specific theory advanced by Vittorio Gallese (2005), and is situated within the sub-discipline of embodied cognition. Embodied simulation assumes a “common functional mechanism is at the basis of both body awareness and basic forms of social understanding” (Gallese 2005, p. 23). Embodied cognition is discussed in depth in Chapter 4 and embodied simulation in Chapter 5.

------this page intentionally blank ------

2.1 Sound and/or Film Studies?

2.1.1 Introducing the task of theorising cinematic sound Should cinematic sound27 be examined first and foremost as sound or as something subsumed by the moving picture? Is cinematic sound best understood under the rubric of Sound Studies or Film Studies28, or something as yet unspecified?

Responding to these questions, the purpose of this chapter is to review the extant literature that is most germane to my particular investigation of the function of cinematic sound design, paying special attention to recent developments in affect studies and neurological approaches to sound design. First we must acknowledge that sound design is a contentious term. This is so both in industry and academia. What exists is an eclectic body of research which theorises ‘film-sound’. This extant theory mostly descends from semiotic, psychoanalytic and Marxist film theory which dominated the 1970s and 1980s, and focuses mainly upon voice and music within narrative cinema. Christian Metz’ The imaginary signifier: psychoanalysis and the cinema (1982), Mary Anne Doane’s ‘The voice in the cinema: the articulation of body and space’ (1980a) and ‘Ideology and the practice of sound editing and mixing’ (1985), Kaja Silverman’s The acoustic mirror: the female voice in psychoanalysis and cinema (1988) and Claudia Gorbman’s Unheard melodies: narrative film music (1987) are prime examples of this body of theory. Focused as it is upon voice and music, this corpus is limited in its observations concerning the third strand of the soundtrack: sound effects. Cinematic sound effects, as shall be discussed below, remain under-explored in the literature. Thus the design of sound effects provides a relatively open and uninscribed channel through which this thesis may propound some fresh propositions and theories about cinematic sound design.

The current chapter is comprised of four sections. The first section (tabulated as 2.1 etc.) introduces general problems and opportunities in the theorising of cinematic sound. The second section (tabulated as 2.2 etc.) engages with the ambiguity of sound design as a term and as a practice. The third section (tabulated as 2.3 etc.), ranging from early through classical to post-classical cinema, surveys the extant literature theorising film-sound. This section engages with the work of two highly cited film-sound theorists – Rick Altman and Michel Chion – to highlight the convictions which

19 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. underpin standard film-sound theory and to show the areas of innovation that still beckon, especially around the design of sound effects. Lastly, the fourth section (tabulated as 2.4 etc.) frames this gap in Film Studies and points to the areas of emerging knowledge that are of special concern in this thesis.

2.1.2 Problems and opportunities for soundtrack scholarship Studies of film-sound have been habitually prefaced with acknowledgements that such scholarship is either scarce, lacking, or a ‘special issue’ (Brophy 2008; Hilmes 2005). If this claim were once accurate it no longer rings completely true (Birdsall & Enns 2008, p. 1; Hanson 2007, p. 30). The first decades of the 21st century evince a proliferation in studies of screen sound (that is, sound and its relationship with the moving visual image) through a broad number of journals, edited volumes and monographs.

The growing depth in the theorising of screen sound is expressed through such journals as Music, Sound and the Moving Image (2007–present), Music and the Moving Image (2008–present), The Soundtrack (2007–2012), The New Soundtrack (2011–present), and SoundEffects (2011–present). Next is the expanding array of edited volumes from significant publishing houses such as Oxford University Press, Routledge, Palgrave, and Continuum. Offerings from OUP include the imposing volumes The Oxford handbook of new audiovisual aesthetics (Richardson, Gorbman & Vernallis 2013), The Oxford handbook of sound and image in digital media (Vernallis, Herzog & Richardson 2013), and The Oxford handbook of film music studies (Neumeyer 2014). The offerings from Routledge are equally rich, and include The Routledge companion to screen music and sound (Mera, Sadoff & Winters 2017), The Routledge companion to music and visual culture (Shephard & Leonard 2014), and The Routledge companion to sounding art (Cobussen, Meelberg & Truax 2017). From the Palgrave Studies in Audio-Visual Culture series, The Palgrave Handbook of Sound Design and Music in Screen Media (Greene & Kulezic-Wilson 2016) and Feeling film: a spatial approach (Carroll, B 2016), The visual music film (Mollaghan 2015), The musicality of narrative film (Kulezic-Wilson 2015), The sounds of silent films: new perspectives on history, theory and practice (Tieber & Windisch 2014) and, from Continuum, Sound and music in film and visual media: a critical overview (Harper, Doughty & Eisentraut 2009). Other significant volumes are Lowering the boom: critical studies in film sound (Beck & Grajeda 2008b) edited by Jay Beck and Tony Grajeda, and Sound: dialogue, music, and effects (Kalinak 2015) edited by Katherine Kalinak, as well as Kevin Donnelly’s The spectre of sound: music in film and television (Donnelly 2005) and Occult aesthetics: synchronization in sound film (Donnelly 2014), Jay Beck’s Designing sound: audiovisual aesthetics in 1970s American cinema (Beck 2016), Giorgio Biancorosso’s Situated listening: the sound of

20 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. absorption in classical cinema (Biancorosso 2016) and Helen Hanson’s Hollywood soundscapes: film sound style, craft and production in the classical era (Hanson 2017). Running alongside film- soundtrack studies is an emerging form of sound theory represented by the work of Mark Grimshaw and colleague Tom Garner (Garner, T & Grimshaw 2014; Garner, TA 2018; Grimshaw 2008; Grimshaw & Garner 2015) in the field of embodied virtual acoustic ecology, particularly in the form interactive media such as computergames29. As these titles attest, the current era is in the midst of a burgeoning exploration of the soundtrack, and its momentum is offsetting the historical balance within film theory skewed toward visually-oriented scholarship.

Simultaneously, the field of Sound Studies has experienced a similar rise in scholarship. Journals such as Organised Sound (1996–present) and Sound Studies (2015–present), blogs such as Sounding Out! (Sounding Out! 2009–present), and edited volumes such as The auditory culture reader (Bull & Back 2003), Audio culture: readings in modern music (Cox & Warner 2004), Hearing cultures: essays on sound, listening, and modernity (Erlmann 2004), Sound Studies: new technologies and music (Pinch & Bijsterveld 2004), Sonic mediations: body, sound, technology (Birdsall & Enns 2008), The Oxford handbook of Sound Studies (Pinch & Bijsterveld 2012), The Sound Studies reader (Sterne 2012), Electrified voices (Zakharine & Meise 2013), Soundscapes of the urban past: staged sound as mediated cultural heritage (Bijsterveld 2013), and Voice: vocal aesthetics in digital arts and media (Neumark, Gibson & Leeuwen 2010) explore the febrile nexus of speech, music, sound art, technology, culture and politics. Jonathan Sterne’s The audible past: cultural origins of sound reproduction (2003), Emily Thompson’s The soundscape of modernity (2004), and Frances Dyson’s Sounding new media: immersion and embodiment in the arts and culture (2009), Seth Kim-Cohen’s In the blink of an ear: towards a non-cochlear sonic art (2009), Brandon LaBelle’s Acoustic territories: sound culture and everyday life (2010) and Background noise: perspectives on sound art (2006), and Salomé Voegelin’s Listening to noise and silence: towards a philosophy of sound art (2010) are just some of the field’s more significant monographs from the past two decades. What these texts share is a focus upon modernity and the relationship between sound, technology and culture in the formation of listening and subjectivity. However, as Sterne observes, for all the knowledge of history and philosophy of sound, the field remains “conceptually fragmented” (Sterne 2003, p. 4).

Maybe this fragmentation is at the root of the field’s apparent difficulty to crystallise into a bona fide discipline? This anxiety echoes repeatedly throughout the literature. Rick Altman, whom Michele Hilmes considers the “godfather of Sound Studies in the United States” (Hilmes 2005, p. 250), called

21 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. for a study of ‘Film sound, all of it’ (Altman 1999a) and prompted Sound Studies to be ‘a field whose time has come’ (Altman 1999c). Similarly, Hilmes once observed sound culture studies hesitates to declare itself (Hilmes 2005), and Lars Nyre suggested ‘Sound Studies is still tuning in’ (Nyre 2010). Even recent overtures for Sound Studies to become a discipline in its own right continue to be controversial, and the generally agreed position seems to be the retention as of its interdisciplinary status as a field rather than as a discipline (Kromhout 2014). Issues of interdisciplinarity are at the heart of the dilemma, and these issues, which shall be discussed at the end of this chapter, have relevance to a theory of cinematic sound.

2.1.3 Sound qua sound (not merely music) The nature and meaning-making function of cinematic sound(-as-sound) – as sound effects, as Foley, as atmospheres – is not the usual focus for analysis in film-sound theory30. A feature of the publications mentioned above is a tendency toward theorising speech and music. (A notable exception is Sound: dialogue, music, and effects (Kalinak 2015) which places comparatively similar weight upon the function and design of sound effects as the other elements of the soundtrack.) That speech and music have thus far received most of the theoretical attention is not surprising. Speech and music are clearly products and modes of human communication and intentionality; the case cannot be made so readily for sound effects which encapsulates a wildly broad category of sound ranging from the atmospherics of water and wind, animal and bird calls, machinery of all kinds, the human movement of Foley, and the spectacular (and impossible) sound of spacecraft screaming through the vacuum of space. Sound collected under this rubric emanates more from the everyday natural world than the world of human intentionality. True, this form of non-speech and non-musical sound has usually been formulated as ‘noise’. But noise is a term laden with negative connotations, and, even when not-so- negative, one cannot help feeling John Cage’s conception of it attempts to rehabilitate this form of sound as ‘musical’31. The implicitly valued bias toward ‘musicalisation’ is similarly apparent in Chion’s claim that “you have to understand music to be able to talk about the use of sound in cinema” (Chion quoted in Warburton 2008).

A recurring theme in cinema sound discourse is the problem of a deficient critical terminology (Altman 1985, 1992a; Chion 1994; Johnson, W 1985, 1989; Sergi 2004). Jonathan Rosenbaum, for instance, notes the “brutal inadequacy” of the term sound effects (1978, p. 38). Another brutal term is noise which connotes the design of sound effects as unwanted, unnecessary and non-aesthetic. The desire of this thesis to theorise the ‘unnecessary’ may seem counterintuitive, but the design of cinematic non-speech and non-

22 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. musical sound is media theory’s ‘blind-spot’, and because of this it offers great promise for exploration and discovery.

2.2 Sound design: history and definition of an ambiguous term

2.2.1 An alternate historiography of sound and cinema Histories of cinema are almost always ocularcentric, but it is possible to advance an alternate historiography. It might, for example, de-emphasise the Talkies and the conversion to synchronous sound as a watershed, and might even downplay the development of the phonograph, preferring instead to trace the origins of mediated sound to 1857 when Édouard- Léon Scott de Martinville patented a device he called the phonautograph (Giovannoni et al. 2008). The phonautograph recorded sonic phenomena as a visual trace of smoke captured upon rag-paper and, as a consequence, blurs an easy technological distinction between mediated sight and sound.32 De Martinville, who was a bookprinter and librarian, conceived of his invention as a stenographic aid, and intended his recordings to be viewed, categorised and archived but never reproduced.33 However, as serendipity would have it, de Martinville’s achievements did, in fact, come to be heard when in 2008 David Giovannoni and Patrick Feaster reproduced sound from a number of phonautograms via a laser stylus (Giovannoni et al. 2008). These phonautograms provide our earliest recorded evidence of a human voice (Giovannoni et al. 2008): an uncanny presence which comes to us from a quite different time, space and place. In 1878, seventeen years after de Martinville’s patenting of the phonautograph, Thomas Edison patented the phonograph. Edison’s intentions for the phonograph stand in stark philosophical contrast to the non-reproducing technology of de Martinville’s phonautograph. From the outset, Edison considered the phonograph to be a technology for both recording and reproducing sound.

Three decades after the patenting of the phonautograph and one decade after the phonograph, Louis Le Prince photographed the Roundhay Garden scene (Le Prince 1888b) and Traffic Crossing Leeds Bridge (Le Prince 1888a) thus providing our earliest extant moving photography34. To confound the interplay between recorded sound and recorded moving pictures further, only a handful of years after Le Prince, William K. L. Dickson filmed the Dickson Greeting (Dickson 1891) and produced the earliest known synchronous sound-film in the Dickson Experimental Sound Film (Dickson 1894/1895), both created while Dickson

23 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. was working for Edison. The Dickson Greeting was produced using a kinetoscope and the Dickson Experimental Sound Film using a kinetophone, a hybrid device which synchronised Edison’s phonograph and kinetoscope.

Contemporaneous with the Lumieres’ L'Arrivée d'un train en gare de La Ciotat (Lumière, A & Lumière 1895) and La Sortie de l'Usine Lumière à Lyon (Lumière, L 1895), Dickson’s invention illustrates, from the beginning, how sound and cinema are intimately interwoven and cannot be stripped apart. Tom Gunning reminds us that Edison considered the kinetoscope to be a device for “doing for the eye what the phonograph does for the ear”, and devices for recording sound and moving pictures to be not only similar in kind but that the phonograph provided a template for understanding cinematography (Gunning 2001, p. 16). Moreover, with reference to Georges Demenÿ, Gunning goes so far as to imply cinematography is conceptually parasitic upon the development of the phonograph when he notes “motion pictures began as an image of sound” (Gunning 2001, p. 14), or, at minimum, “motion photography and the phonograph implied each other” (Gunning 2001, p. 16). Nevertheless, in spite of de Martinville’s pioneering sound technology emerging thirty years before Le Prince's moving pictures, and Dickson’s synchronised sound-film appearing immediately before the Lumières’ celebrated presentation of silent moving pictures35, it is revealing that Sound Studies, as Michele Hilmes wryly observes, has been “hailed as an ‘emerging field’ for the last hundred years” (Hilmes 2005, p. 249). The conundrum of sound’s status within theory suggests it is subject to veiled assumptions. Hilmes, like Sterne (Sterne 2003, p. 15), proposes some of the difficulties encountered when theorising sound emanate from sound’s association with emotion and subjectivity pitched against the supposed objectivity and rationality of vision (Hilmes 2005, p. 249). At play are dualistic notions which bifurcate vision from audition and thought from feeling.

What explains the relative ‘deafness’ in the academy (Brophy 2008)? Lodged deep in the West’s philosophical and cultural heritage is a presumption that only vision constitutes the most compelling or legitimate form of knowledge (Levin 1993a). In contrast to the long lineage of ocularcentrism that runs from Plato’s and Aristotle’s linking of knowledge and reason to vision through to the rise of Cartesian dualism during the Enlightenment, it is only relatively recently that studies such as David Michael Levin’s edited volume Modernity and the hegemony of vision (Levin 1993b), Martin Jay's Downcast eyes: the denigration of vision in twentieth-century French thought (Jay 1993), the work of Walter Ong (Ong 1991) in

24 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. language and knowledge, even Juhani Pallasmaa’s condemnation of ocularcentrism’s influence over architecture in The eyes of the skin: architecture and the senses (Pallasmaa 2005), have begun to prosecute a critical re-examination of vision in the formation of knowledge of the world.

Thus it is not surprising that the standard paradigm for understanding cinema has long been that of visual narrativism. Visual narrativism is produced by the interlocking logics of ocularcentrism and narrativism. Ocularcentrism, for example, is epitomised in the thinking of two theorists who were crucial in the development of cinema studies, Rudolf Arnheim and André Bazin. Arnheim rejected sound-film as aesthetically inferior to the ‘purity’ of silent cinema whilst for Bazin cinema bore a unique indexical relationship to reality because of its visual trace in photography. Arnheim’s and Bazin’s predispositions led to an ever- strengthening emphasis, in classic Twentieth Century film studies, on narratively organised, visually-witnessed knowledge. The rise of narrativism may be traced through the work of influential analysts of myth and fabulation such as Vladimir Propp (Propp 1968), Roland Barthes (Barthes 1977a) and Aldirgas Greimas (Greimas 1983), as the ‘narrative turn’ within the academy made explicit many assumptions already embedded within film theory; especially the notion that narrativity is the primary paradigm by which meaning is created in film cultures the world over36. Narrativism is on full display, for example, when Christian Metz asserts film to be innately narrative (Metz 1975), and when Barthes makes claims about the primacy and cross-cultural universality of narrative (Barthes 1977a). Like vision in relation to the other senses, narrative theory has tended to expand to fill the available space for discourse such that a vast bulk of human knowledge becomes represented in terms of stories (Brewer 1995; Kindt 2009; Kreiswirth 1992). What appears at stake is that human understanding of the world depends on narrative. Taken together, the assumptions of ocularcentrism and narrativism have thus inevitably had an emphatic influence over the status of sound within Film Studies.

Attending to the new generation of scholars who are revising and even resisting an ocularcentric philosophy of sound, I have taken guidance from Casey O’Callaghan (2007). O’Callaghan shifts the argument from a unimodal visual stance to a decidedly multimodal one when he no longer considers vision as metonymic for all human sense modalities. Work such as O’Callaghan’s helps us understand how – when we acknowledge human perception as fundamentally multimodal – a new constellation of relationships opens up for consideration.

25 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Where once was predominantly a ‘pure’ vision structuring the world in a rational and narrative process there is now a matrix of crossmodal and affectively-laden interactions granting cognition of an interpretable world.

Building on these revisions of ocularcentrism and narrativism, deepening the investigation into multimodalism, Chapters 3 and 4 of this thesis will go on to examine in greater detail the multimodal nature of human sense-making, the possibility of a neuroaesthetic production of meaning, and its relevance for a theory of cinematic sound, and in Chapter 6 the neurophenomenological relationship is investigated between cinematic imagery, narrative and conscious thought.

2.2.2 Sound design: history of an ambiguous term Sound design is a highly contested term associated with the rise of ‘New Hollywood’. It is synonymous with Apocalypse Now (Coppola 1979) and the work of Walter Murch.37 Of his intention to re-imagine film-sound as an integral element of the cinematic experience, Murch says:

[That] was the Zoetrope dream at the beginning – the whole concept of what turned into the sound designer in the Zoetrope sense – which is a director of photography for sound. Somebody who took on the responsibility of ‘auralizing’ the sound for the film and making definitive, creative decisions about it. Someone the director can talk to about the total sound of the film the way he talks to the cameraman about the look of the film. (Kenny 1998, p. 20)

Over time, sound design has not retained such a singular definition, and currently possesses at least three possible variations. The first and original definition refers to the overall aesthetic design of all soundtrack elements. The chief exemplar, of course, is Murch’s own work in Apocalypse Now (Coppola 1979). Sound design, in this sense, seeks to parallel cinematography in its elevation, authority and concentration of creative practice. The second definition refers to the design of high-value sound effects (SFX) for specific objects, actions or events. Exemplars might include Murray Spivack’s creation of Kong’s vocalisations for King Kong (Cooper & Schoedsack 1933), Ben Burtt’s lightsaber sounds for Star Wars (Lucas 1977), or Gary Rydstrom’s T-rex roars in Jurassic Park (Spielberg 1993). Sound design in both these definitions is essentially a form of sound composition38 but its materials are drawn

26 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. principally from environmental sources rather than the readymade palette of musical instrumentation. The third and most recently emerged (albeit minor) definition of the term refers to a process by which sonic qualities are programmed into software-based virtual instruments. Exemplars of this kind of computer-based music programming language are Pure Data (Puckette 1996) and MAX/MSP (Puckette 1997). Taken together, these three descriptions give the concept of sound design an ambiguity and flexibility which responds to the context in which it is used but which nevertheless points to an underlying although undefined set of qualities.39

Sound design and sound designer are commonplace terms which remain imprecise within contemporary industry and scholarship. This imprecision has led some to suggest the designations to be merely “fashionable” (Corbella 2010, p. 1). Corbella attempts to undermine sound design as an umbrella term by which multiple aesthetic, conceptual and technical processes may be unified when he claims “unlike the director, the sound designer has not always existed” (Corbella 2010, p. 2). Such a claim oversimplifies the situation. The design of sound has long been a practice, but a practice without name.

Apocalypse Now is acknowledged to be the first movie credited with ‘sound design’ but the concept and practice had long been in gestation at American Zoetrope.40 Prior to Apocalypse Now, Murch is credited with ‘sound montage’ on The Rain People (Coppola 1969), THX 1138 (Lucas 1971), American Graffiti (Lucas 1973), The Conversation (Coppola 1974a) and The Godfather: Part II (Coppola 1974b), and George Lucas had engaged Ben Burtt to create a vast and original library to sound the alternate universe of Star Wars (Lucas 1977).

The practice of sound design is often associated with cinematic spectacle and is a conspicuous characteristic of New Hollywood thus suggesting an industrial and historical specificity. However, this too is an oversimplification. Sound was already being ‘designed’ in locations other than the United States. Jim Shields, for example, was working concurrently in Britain to create the extraordinary soundscapes of Alien (Scott 1979). Sound design also occurred outside the Anglophone world, and one need only think of the sound of the movies directed by Alberto Cavalcanti, René Clair, Robert Bresson, and Satyajit Ray. And in the USA sound was also demonstrably ‘designed’ before the emergence of New Hollywood. The work of Arthur Kirbach serves as illustration. In both Possessed (Bernhardt 1947) and A Letter to Three Wives (Mankiewicz 1949) Kirbach explored the stylistic potential of sound to extend

27 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. cinematic grammar and develop narrative. In A Letter to Three Wives (Mankiewicz 1949), Kirbach blended the voice of the movie’s unseen narrator, Addie Ross (Celeste Holm), with diegetic sound effects and extra-diegetic music to produce a series of aural transitions which trigger character flashbacks.41 These transitions are ripples in sonic rhythm that not only signal a character’s stream-of-consciousness but indicates to the audience a shift in narrative time. Kirbach’s technique used a Sonovox,42 a forerunner to the vocoder, to modulate a sound by a secondary waveform, and remains a surprisingly contemporary technique.

Prior to Kirbach is Murray Spivack’s exceptional work in supervising the sound of King Kong (Cooper & Schoedsack 1933), especially the multi-track compositional techniques that he applied to the creation of Kong’s ‘signature’ roar (Boone 1933, p. 21). The guiding principles and techniques of sound creation for King Kong are strikingly similar to the digital sampling techniques used 60 years later by Gary Rydstrom for Jurassic Park (Spielberg 1993). The sophistication of Spivack’s sound is even more astonishing when one considers the release of King Kong to be only a few years after Hollywood’s transition to synchronous sound.43

How may the manifestation of such rapid sophistication in movie sound in the 1930s be explained? The answer customarily centres upon Hollywood’s recruitment of creative personnel from radio. However, the career of Ora Nichols reveals even this to be an oversimplification. Nichols was in charge of sound effects at CBS Radio in New York during the 1930s and 1940s, and supervised many long-running series such as The March of Time (1931–1945) and Buck Rogers in the 25th Century (1932–1947) (Mott 1993, pp. 14, 19, 21- 26; Verma 2012, pp. 45, 47). Nichols’ sound praxis is distinctly that of a ‘sound designer’; she evokes powerful images in the mind of her audience by manipulating the qualities of sound effects. Indeed, it was Nichols who supervised the sound effects for Orson Welles’ infamous radio drama The War of the Worlds (Welles 1938). Much has been made of Welles’ cinematic experimentation with a ‘radio aesthetic’44, but it was Nichols’ understanding of sonic texture, perspective and metaphor which provided the foundation upon which Welles built. Nichols’ praxis, however, did not develop in radio. It developed in silent cinema. And before that vaudeville45. What Nichols’ wielded was a vast tacit knowledge46 of sonic imagery capable of traversing from stage to cinema through radio and back into cinema47. In this way, the praxis of sound design may be said to have pre-existed cinema. Certainly, it was not invented in ‘New Hollywood’. From a film-theoretical perspective, therefore, we clearly need

28 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. more explicit knowledge of the implicit rules which have long governed the transmedial practice of sound design.

2.2.3 Sound design: definition of an ambiguous term The sound of a movie is not merely captured but constructed. Indeed, all aspects of the soundtrack are subject to design, and this process is highly diffused (Whittington 2007, pp. 93-114). This diffusion frustrates any desire to reduce the process to a single authorial individual. Moreover, sound design is not necessarily restricted to the post-production phase as commonly believed; it is a distributed process undertaken by a broad cohort of creative personnel with differing responsibilities and competencies across multiple phases of production and post-production. The chain of sound design spans the production phase and the decisions made by the location sound team (typically comprised of recordist and boomswinger) through to the post-production phase and the impact of choices made by the supervising sound editor, dialogue and ADR editors, sound effects editors, Foley artists and recordists, music editors, and re-recording mixers. The process of sound design occurs through the accretion of many, many decisions, both small and large, which are made with or without an appointed ‘sound designer’48. I acknowledge the idiosyncrasy of this description; however, it is important to recognise the contribution each practitioner brings to bear upon the design of the final soundtrack. Admittedly, this description fails to capture the contribution of the picture editor49, the music composer and her team, nor the preferences of the director, producer and executive producers. Making an even more radical proposition is Randy Thom (2011) who suggests sound design commences with scriptwriting.

In sum, and the cardinal point for the purpose of this thesis, is that sound design is a complex and distributed creative endeavour. Cinema sound is a tacit design practice performed by a wide variety of creative personnel often without shared overt instruction. Furthermore, it is active in both narrative and non-narrative cinema, and is transmedial in nature.

In Chapter 1, I loosely defined sound design as a general creative process by which sound fragments are created, selected, organised, and blended into perceptually unified and coherent ‘sound objects’. Definitions of cinema sound generally oscillate between the micro of the “creation of specific sound effects” and the macro of the “conceptual design of the overall sound track” (Whittington 2007, p. 3). However, these descriptions do not deliver a definition of the purpose and function of sound design50. If the above describes the who and

29 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. how of sound design there remains the definition of what sound design designs and why. On this Sandra Pauletto (2012) observes:

Sound design describes the idea that the sound heard when we experience a film, even when seeming very naturalistic, has been constructed and highly manipulated in order to communicate information – about the characters, their emotional state and intentions and about the places they inhabit – in an aesthetically and stylistically appropriate manner. (Pauletto 2012, p. 129)

For Pauletto, the function of sound design is to “communicate information” about characters’ intentions and emotions as well as about the environment51. Leaving aside the issue of what constitutes an “aesthetically and stylistically appropriate manner”, Pauletto’s definition advances a conventional narrative model which privileges the understanding of character through action and reaction. This is an uncontroversial function within narrative cinema but does not provide a granular explanation as to how the textured materiality of sound achieves such narrativity. Additionally, explanations should be developed to explain how sound functions within non-narrative episodes operating within narrative cinema, as well as non- narrative cinema in toto. What is needed, then, is specificity of the design mechanisms by which auditory information comes to represent or elicit affect, how affect is diffused through the prism of character comprehension and identification or registered directly by the audience, as well as aesthetically trafficked as style.

2.3 Extant literature theorising film-sound

2.3.1 Types of extant film-sound literature A number of methodological trends emerge when reviewing the extant film-sound literature. These trends have been usefully classified into four general types by Amy McGill (2008, pp. 16–17). Within the first type – scholarly research focussed upon theoretical approaches to the soundtrack – McGill includes semiotic, feminist and psychoanalytic theory such as that developed by Kaja Silverman (1988), Amy Lawrence (1991) and Caryl Flinn (1992), as well as the studies of form by Michel Chion (Chion 1994, 1999, 2009, 2016), and musicological studies such as those by Claudia Gorbman (1987) and Royal S. Brown (1994). Additionally, this group may be extended to include the cognitive theories of filmic sound such as those

The second type – scholarly research focussed upon the contexts of soundtrack production – examines professional practice and technologies, usually from historical, industrial and cultural perspectives, and are exemplified in the work of Rick Altman (1992e), James Lastra (2000), Gianluca Sergi (2004), Mark Kerins (2010), William Whittington (2007) and Robert Spadoni (2007). Edited volumes such as Weis & Belton’s Film sound: theory and practice (1985) and Rick Altman’s Sound theory/sound practice (1992e) are foundational in establishing an Anglophone study of film-sound. A variant of this type is the model developed by Philip Brophy in his Cinesonic International Conference on Film Scores and Sound Design series, the proceedings of which have been published as three volumes (Brophy 1999, 2000, 2001). This model, whereby theorists and practitioners interact in a conference setting, has been emulated by Sider, Freeman & Sider’s School of Sound (2003).

The third type – theoretical reflections written by practitioners taken up by scholars – brackets work written by film-sound practitioners which are subsequently analysed or referenced by Film Studies scholars. Eisenstein, Pudovkin & Aleksandrov’s ‘The sound film: a statement from the USSR’ (1928), Pudovkin’s ‘Asynchronism as a principle of film sound’ (Pudovkin 1985 [1929]) and Robert Bresson’s Notes on the cinematographer (1986) are archetypal examples of this form of praxis-generated theory. Strong contemporary examples of sound practitioners reflecting directly upon the aesthetic and technical dimensions of their craft are Walter Murch’s In the blink of an eye: a perspective on film editing (1992), ‘Stretching sound to help the mind see’ (2000) and ‘Dense clarity – clear density’ (2005) and Randy Thom’s ‘Designing a movie for sound’ (1999), ‘On sound designing: Cast Away’ (2009), ‘Screenwriting for sound’ (2011) and ‘Notes on sound design in contemporary animated films’ (2013). An auxiliary to these texts are Vincent LoBrutto’s Sound-on-film: interviews with creators of film sound (1994) and Nicholas Pasquariello’s Sounds of movies: interviews with the creators of feature sound tracks (1996) both of which contain interviews with significant figures in North American film-sound such as Richard Beggs, Ben Burtt, Cecelia Hall, Skip Lievsay, Mark Mangini, Gary Rydstrom, Leslie Shatz, and Frank Warner.

The fourth type – artistic and technical instructional texts written by sound practitioners for sound practitioners – includes texts like trade journals such as Journal of the Audio

31 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Engineering Society (1948–present), Mix (1977–present), and Sound on Sound (1985–present) but also instructional volumes like David Sonnenschein’s Sound design: the expressive power of music, voice, and sound effects in cinema (Sonnenschein 2001) and David Yewdall’s Practical art of motion picture sound (Yewdall 2003). Additionally, another variant has emerged with the advent of media-rich online resources. Sven Carlsson’s Filmsound (1996) is arguably the earliest example of such an online resource. SoundWorks Collection, produced by Michael Coleman (2009), provides access to sound professionals discussing their craft by taking full advantage of the online capacity to deliver interviews and expositional materials focussed upon high-end movie, television and computergame production. Not only is this material available through a dedicated website but these audiovisual resources are distributed as podcasts and via platforms like SoundCloud, Vimeo and YouTube. Designing Sound (2008) boasts a broad stable of contributing authors, and a deep archive of interviews, reviews, tutorials and links to other resources. Additionally, the social web has enabled sound design practitioners to directly seek guidance from other practitioners. Social Sound Design (undated) and Sound Design Stack Exchange (2010) are communities built upon the ability to share insights and generate commentary, and the exchange of ideas and know-how is vigorous. These online sites present a vast reservoir of implicit knowledge which, as far as I am aware, is yet to be mined for its embedded conceptual content. I labour this point because it speaks directly to one of the four planks identified by Mark Kerins (2008) when proposing a contemporary research agenda for cinema sound. Kerins is critical of the analytical models which have been imported into Film Studies, and urges focus upon the praxis (and hence tacit knowledge) of sound practitioners so that the explication of this tacit knowledge can contribute to the development of “a truly filmic model of sound studies” (Kerins 2008, p. 115).

In summary, the above methodological trends constitute distinct modes of address shaped by (i) concepts developed by scholars for scholars, (ii) concepts developed by (or derived from) practitioners and assimilated by scholars, and, lastly, (iii) concepts developed by practitioners for practitioners. While these modes constitute a broad ecology for understanding cinematic sound design it also indicates boundaries where friction as well as absences can occur when forming a comprehensive, cohesive and commonly held set of understandings about cinema sound.

32 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Thus far I have noted how the study of film-sound theory overlaps disciplines, sketched an alternate history of cinema which is not wedded to visual primacy, and provided historical background and definitional constraints upon the practice of sound design. In the following section, the conceptual landscape of film-sound theory is periodised into three phases – early cinema (1895–1919), classical cinema (1917–1960), and post-classical cinema (1960– present).52 These periodisations do not result from hardened rules but acknowledge technological, industrial, aesthetic, philosophical, social and cultural aspects of filmic production and consumption do not march in lockstep but unfold in a loose and co-defining synchrony. Film-sound theory tends to reverberate with the preoccupations echoing through Film Studies, and so each of these subsections focuses upon film-theory generally and film- sound theory specifically. The purpose of this organisational structure is to describe the milieu of dominant intellectual forces within which film-sound theory emerged, and to understand how these forces facilitated or inhibited the development of sound theory.

2.3.2 Early sound and cinema (1895–1919) Early cinema is conventionally circumscribed between 1895 and 1919, and brackets the Lumiére’s premiere at the Salon Indien du Grand Café through to the emergence of the Hollywood studio production system. The conventional historiography of writings generated within this period typically features the work of Robert Grau, Vachel Lindsay, Terry Ramsaye, and Hugo Münsterberg. Robert Grau was an entertainment journalist with a belief in ‘technological progress’, and produced one of the first techno-histories of the film industry, The theatre of science: a volume of progress and achievement in the motion picture industry (Grau 1914). Around this time, Vachel Lindsay, who was building a reputation as a poet of consequence, published possibly the earliest Anglophone example of film aesthetics and criticism, Art of the moving picture (Lindsay 1915). A decade later, Terry Ramsaye, an engineer-turned-journalist-turned-film-producer, published an historical overview of the early years of cinema, A million and one nights: a history of the motion picture through 1925 (Ramsaye 1926).

A prescient and provocative stranger amongst even this eclectic group is famed German psychologist Hugo Münsterberg53. Invited by William James in 1892 to supervise an experimental psychology laboratory at Harvard University, Münsterberg developed a deep fascination with cinema during his time in the United States. Unsurprisingly for an experimental psychologist, Münsterberg was determined, firstly, to explore how “the

33 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. photoplay obeys the laws of the mind” (Münsterberg quoted in Langdale 2002, p. 91). Specifically, he pondered how filmic images engage the basic psychological functions of perception, attention, emotion, memory and imagination during film reception. Münsterberg considered cinema to capture its audience by ‘mimicking’ these psychological functions to present “as-if perceptions, as-if thoughts, as-if streams of associations, and as-if emotions” and thus to “display subjectivity” (Tan, ESH 2018, p. 2). Effectively, in The film: a psychological study; the silent photoplay in 1916 (1916) Münsterberg proposed that cinematic processes are analogue to processes of the human mind54. This was an aesthetic as well as a psychological account of film, with the phenomenological emphasis of what it is like to experience a movie. This link between cinema and human psychology and neurology – presciently investigated by Münsterberg – would prompt and reward analysis for the next century (Tan, ESH 2018). It is not controversial to observe, though, that the psychology and neurology of cinematic vision received earlier and more thorough theoretical attention than did the psychology and neurology of audition.

On Munsterberg’s account, film was laudable so long as it was produced and consumed as an artform rather than as popular entertainment. The attainment of artistic standards depended on cinema’s visual purity, he maintained, and if adulterated by sound – performed live or, more threateningly, by the emerging technologies for synchronous sound film – cinema failed to meet the necessary criteria (Münsterberg in Langdale 2002, pp. 144–145). Cinema, Münsterberg maintained, “cannot gain but only lose if its visual purity is destroyed” (Münsterberg in Langdale 2002, p. 145). In Münsterberg’s visual definition of cinema we can discern the aesthetic concept of ‘medium specificity’, a concept traceable to Gotthold Lessing’s Laocoön: an essay upon the limits of poetry and painting (1874 [1766]). Medium specificity theory argues each artform is constituted and differentiated by unique material qualities of the medium which have been established and conventionalised. Münsterberg took the visual image to be the primary characteristic of cinema, and argued against sound film as aesthetically ill-founded. Thus, Münsterberg considered the escalating number of experiments in synchronous sound to be a threat to the visual purity of film, particularly the synchronisation of phonographic speech (Langdale 2002, p. 23). He was, however, more accommodating of film-music. Although not an integral part of the film experience, on Münsterberg’s account music “relieves the tension and ultimately keeps the attention awake” even as it “must be entirely subordinated [to the pictures]” (Münsterberg quoted in Langdale 2002, p. 146). Regarding sound effects, however, Münsterberg was openly hostile. Sound

34 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. effects, he argued, “have no right to existence in a work of art that is composed of pictures” (Münsterberg quoted in Langdale 2002, p. 146), and held particular disdain for sound effects created through the use of a machine called an Allefex55:

[B]etween the intolerable accompaniment by printed or spoken words on the one side and the perfectly welcome rendering of emotionally fitting music on the other, we find the noises with which the photoplay managers like to accompany their performances ... They even have a complicated machine, the "allefex," which can produce over fifty distinctive noises, fit for any photoplay emergency … They [sound effects] have no right to existence in a work of art which is composed of pictures. In so far as they are simply heightening the emotional tension, they may enter into the music itself, but in so far as they tell a part of the story, they ought to be ruled out as intrusions from another sphere … The limitations of an art are in reality its strength and to overstep its boundaries means to weaken it. (Münsterberg quoted in Langdale 2002, p. 146).

Münsterberg’s animosity to sound film sets the tone for classical film theory, and his defence of cinema as a visual art is recognisable and amplified in Arnheim’s ‘A new laocoön: artistic composites and the talking film’ (1938).

While Münsterberg’s modelling of film as an externalisation of the human mind is a powerful insight, his disqualification of auditory perception from that modelling is a curious deficiency that indicates an underlying ocularcentrism. Nevertheless, notwithstanding the neglect that befell his writing when US-German relations deteriorated during World War I, Münsterberg’s pathfinding theories resonate in this thesis because they brought early attention to the function of perception and emotion in cinematic experience. Münsterberg held “the core of man lies in his feeling and emotions” (Münsterberg 2002 [1915]), and the central aesthetic function of cinema is “to picture emotions” (Münsterberg quoted in Langdale 2002, p. 99). Like William James’ own work in establishing a somatic theory of emotion (1884), Münsterberg conceived of the emotions as embodied phenomena (Tan, ESH 2018, p. 15).

Notably, James suffered an eclipse and posthumous ascension comparable to Münsterberg’s. James’ work fell by the wayside as behaviourism came to dominate the field of psychology, and would have to wait for the emergence of an entirely different generation of theorists in Robert Zajonc, António Damásio, Joseph LeDoux, Paula Niedenthal, and Piotr Winkielman

35 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. before emotion became once more a respectable target for scientific inquiry. Indeed, as we will see in Chapter 4, the James–Lange theory of emotion is foundational in the development of contemporary theories of embodied cognition. Thus, as the work of Münsterberg and James attests, even before film theory was a defined field of inquiry, cinema was demonstrably congenial to scientific investigations of aesthetic emotion56. Although their influence was to lie latent for decades, Munsterberg’s and James’ writings primed the eventual appreciation of cinema as a complex system of aesthetic, psychological and neurological affects and effects. We can understand this potency was recognised during the earliest phases of cinema history, even though overshadowed for many decades by the ascendence of ‘classical narrative’ formats in industrially-scaled cinema, particularly in Anglophone cultures.

Early cinema and its attractions Early cinema was once thought an unremarkable and primitive era, but that account came to be heavily revised as a consequence of the 1978 congress of the Fédération Internationale des Archives du Film (FIAF) held at Brighton, England. Confronted with over five hundred rarely exhibited prints, those film historians in attendance began to question the received thinking about early cinema57. The ‘primitive cinema’ of the late 19th and early 20th centuries was not, as was previously believed, an homogenous era whose function was to act as a rehearsal for classical – that is to say, narrative – cinema58 (Musser 1994a, pp. 203–204). Tom Gunning has shown how early cinema may be periodised into, at minimum, three sub-periods (1986, pp. 66-68). The first, 1895–1906, is characterised as a “cinema of attraction” because it places emphasis on “display rather than storytelling”. Within this era Gunning identifies the years 1903–1906 as the emergence of a second phase, the “synthesis of attractions and narrative” which are epitomised by the chase film59. The chase film, Gunning contends, is “the original truly narrative genre of the cinema, providing a model for causality and linearity as well as a basic editing continuity” (1986, p. 68). Then came the third of Gunning’s sub-periods, 1907– 1913, which “represents the true narrativization of the cinema, culminating in the appearance of feature films”60 (1986, p. 68, emphasis in original). (Straddling these sub-periods, roughly 1905–1909, is the Nickelodeon boom.) In addition to Gunning’s periodisations are the years 1914–1919 which are notable for the application of a Taylorist production model to moviemaking which prepared the way for the Hollywood studio system.

36 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Far from being an unsophisticated, merely preparatory bloc, cinema’s early years are better understood as a tumult of smaller movements and tendencies which bubble with experimentation61. Two significant concepts arising from the post-Brighton revisionism are ‘cinema of attractions’ and ‘monstration’. The development of the term cinema of attraction (later cinema of attractions) is credited to both Tom Gunning and André Gaudreault62. In contrast to the voyeuristic aspects of narrative cinema identified by Metz, the cinema of attraction is “a cinema that bases itself on … its ability to show something” (Gunning 1986, p. 64, emphasis in original). The cinema of attraction emphasises what Gaudreault calls monstration (Gaudreault 1987). As Wanda Strauven relates, “monstration (showing) is to narration (telling) what presentation is to representation” (Strauven 2006a, p. 15) thus marking a central characteristic radically different in nature to classical cinema. The cinema of attraction, as a mode of address, is concerned less with narrative integration than with the qualities of presence.

For Gunning, therefore, the “history of early cinema, like the history of cinema generally, has been written and theorized under the hegemony of narrative films … early cinema was not dominated by the narrative impulse that later asserted its sway over the medium” (Gunning 1986, p. 64). The assumption of narrative primacy, then, hinders more catholic theory-making because it precludes the possibility of other hypotheses. Conceiving of early cinema as a cinema of attractions provides, for Gunning, a more robust understanding of the affordances and potentialities of cinema (Gunning 1993, p. 4).

Gunning purposefully borrows the term ‘attraction’ from Sergei Eisenstein’s concept of “montage of attractions”63 (Eisenstein 1947 [1923]). With this concept Eisenstein sought to undermine the realism of representational theatre, and to achieve this revolutionary aim he considered an ‘attraction’ a suitable “unit of impression” (Gunning 1986, p. 66).

An attraction aggressively subjected the spectator to "sensual or psychological impact." According to Eisenstein, theater should consist of a montage of such attractions, creating a relation to the spectator entirely different from his absorption in "illusory imitativeness." … Then as now, the "attraction" was a term of the fairground, and for Eisenstein and his friend Yuketvich it primarily represented their favorite fairground attraction, the roller coaster… (Gunning 1986, p. 66)

37 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. What is significant here is Eisenstein’s conception of an attraction as a packet of affective experience, something sensual and forceful. Attractions produce astonishment and awe felt in the sensorium of the perceiver64. More specifically, attractions create affect not from narrative absorption but by the immediately felt and emphatically present sensory qualities of the attraction itself. (Clearly, Eisenstein’s fascination for the rollercoaster is telling.)

Thus it is clear that, potent within cinema’s early stages, two modes are equally available for affecting the perceiver: one via presentation and cinematic attractions and the other via representation and narrative integration. These modes which I label perceptual and narrative immersion, respectively, are examined more closely in Chapter 3. However, it is important to recognise Gunning does not pitch cinematic attractions against narrative integration in a zero- sum game (Gunning 1986, p. 70). Indeed, on Gunning’s account, even as cinema moved into its classical narrative phase, attractions were never totally suppressed nor replaced by narrative protocols (Gunning 1993, p. 4).

As will become evermore evident throughout this thesis, of all the affordances proffered by the emerging technologies of early cinema only some were actualised by the classical era; others are now being more fully deployed in contemporary post-classical cinema and interactive media. What attracted, mesmerised and sustained those many inventors, producers and audiences of early cinematic media before it was classically and narratively institutionalised as ‘the cinema’, was the sheer phenomenality of felt experience or presence intensified via the perceiver’s engagement with the cinematic apparatus. Presence can be defined, for the purposes of this thesis, as the feeling that binds us in the lived moment, and has little to do with representation or narrative per se. It is, to use Gunning’s phrase, the “smack of the instant” (Gunning 1993, p. 11).

Post-Brighton scholarship challenged the received knowledge which positioned early cinema as pre-cinema in the mistaken belief it was little more than a rehearsal for narrative cinema. The concept of cinema of attractions challenged the bias of narrative primacy within standard film theory yet the assumption of visual primacy was allowed to pass unchallenged.

Sound design praxis of early cinema If the Brighton FIAF conference was a watershed in discovering the role of attractions in early cinema, the 5th International Domitor conference in Washington, D.C. in 1998 initiated a

38 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. similar sea-change for the role of sound in early cinema65. The conference was organised around the themes of synchronous sound apparatus, sound practices and reception in early exhibition, sound and ethnicity, and the social context of sound in early cinema66. Rick Altman, Richard Abel, André Gaudreault, Tom Gunning and Stephen Bottomore were in attendance, and a highlight of the conference was the first public presentation of the hitherto lost Dickson Experimental Sound Film67 (Dickson 1894/1895).

From the alumni of Domitor 1998, Rick Altman (Altman 1996, 2004, 2012, 2017) has become perhaps the most influential in the way his research has gone on to chart early cinema as, contrary to received opinion, a whirlwind of sound:

In 1908, theaters in the same block on the same day might well offer varying combinations of silence, sound effects, voices behind the screen, a lecturer, a synchronized sound-on-disc Cameraphone projection, a phonograph cylinder or disc, and live music. (Altman 1996, p. 690)

The heterogeneity of techniques listed by Altman betrays the soundscape of pre-1910 cinema to be a collision of diverse vocal, musical and sounding practices. The sound of early cinema was eclectic and noisy. If early cinema was a cinema of attractions, the commotion of sound was one of those attractions.

Contrary to Münsterberg’s assertions about visual purity, silent film provoked a deep apprehension in the movie-going public. The evidence indicates that audiences felt silent moving pictures to be somehow incomplete and unnatural. Eadweard Muybridge instinctively knew this and accompanied the public presentation of his zoopraxiscope studies of locomotion with the beat of a hammer marking the moment his subject’s foot contacted the ground (Bottomore 1999, p. 486). Maxim Gorky was profoundly disturbed by the “soundless spectre” of the Lumière brothers’ demonstration at the 1896 Nizhniy Novgorod Fair (Gorky quoted in Popple & Kember 2004, pp. 2–3). Indeed, by 1909 it was commonplace to read in trade journals of the requirement to both see and hear with directives to exhibitors to end the “silence of death” (Bottomore 2001, p. 130). Perhaps it was an untheorised understanding, but it was a common understanding nevertheless, that when sound is co-presented with moving pictures the experience is potent, exciting and somehow compellingly attractive.

39 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Pertinent to this issue of the managed relationship between vision and audition, it is notable that the latter half of the 19th century delivered a kind of technological divisioning of the human sensorium. Gunning, for example, considers the phonograph to be the originating technology of this destabilisation because of the way it “separated the human senses divorcing ear from eye” (Gunning 2001, p. 16). In a complimentary fashion, as Edison observed, the kinetoscope could “do for the eye what the phonograph does for the ear” (Edison quoted in Gunning 2001, p. 16). But just as notably, there is strong evidence that Edison’s creative impulse in developing motion picture technology was to bring about the effective re- unification of sight and sound by developing a means of conjoining phonographic and kinetoscopic technologies. This stands in stark contrast to Münsterberg who rejected “[t]hose who, like Edison, had a technical, scientific, and social interest but not a genuine aesthetic point of view in the development of the moving pictures” (Münsterberg quoted in Langdale 2002, p. 146). Over time, Münsterberg’s belief that the technical problem of synchronisation was the symptom of an underlying aesthetic misconception appears to be the lesser accurate assessment of the forces driving early cinema. Armed with a theory of art focussed on medium specificity, what Münsterberg missed was the prospect of an aesthetics based upon crossmodal interaction. The cinematic impulse toward the splitting, realignment and reintegration of vision and audition counts as one of early cinema’s attractions, and, in some ways, its earliest engine for meaning-making. Although Gunning pursues only a visual approach when elucidating the cinema of attractions’ ability to produce a “flicker of presence”, it is not unreasonable to also include the audial. Like the visual, sound can produce the “cinematic smack of the instant” (Gunning 1993, p. 11).

Therefore the first audiences of cinema – that is, the audiences of the cinema of attractions – intuitively understood the need for sensory holism. Altman summarises well:

the early cinema soundscape was heavily slanted toward audiovisual matching. Sounds that evoked images were highly prized, as were images that suggested sounds … early cinema audiences harbored expectations of a connection between the audio and the visual portions of the program. (Altman 2017, p. 190)

Such an observation finds explanatory support in perceptual psychology, for example in the studies by James J. Gibson (1966) and Lawrence E. Marks (1978), which propose the human must be construed as a unified perceptual system. The cognitive sciences, therefore, offer

40 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. important insights both for understanding the perennial characteristics of aestheticised sound and for propounding a theory of sound design.

Early sound praxis implies a tacit theory of film-sound The intricacies of the integrative approach and the multimodal interactions that can occur in the fabrication of cinematic imagery – visual and sonic imagery – will be discussed in greater detail in Chapter 3. But for now, maintaining focus on the ‘pre-history’ of sound design and ‘sound theory’ during the early decades of cinema history, we can note how Stephen Bottomore (Bottomore 1999, 2001) has mapped an evolving relationship between early film’s audience and sound effects. Referencing trade publications and reviews from the era, Bottomore parses this period into early, middle and late phases. The early phase, existing from the mid-1890s until around 1906, is characterised by a somewhat non-uniform practice of co-presentation of live sound effects and moving pictures (Bottomore 1999, p. 485). Sound effects were already a potent component of public entertainments from at least the 1840s when soundscapes were performed for cycloramas such as those at London's Colosseum, and this mode of sound production carried over into the large-scale magic lantern shows of the 1850s. These shows not only included scripted sound effects but required the effects to be synchronised against specific visual cues (Bottomore 1999, pp. 485–486). By the time of film’s emergence in the 1890s its audience was well-acquainted with ‘sounding’ pictures, both live or as phonographic rendition of speech, music and sound effects, sometimes within the same performance.

During the middle phase, which existed from approximately 1906 to 1913, a burgeoning number of venues solely dedicated to exhibiting moving pictures coincided with a surge in narrative film production (Bottomore 2001, p. 129). The scale of the phenomenon was immense, as McKernan (2007) has shown, and the use of sound effects proliferated. By chronicling the trade journals’ response to this rapid expansion, Bottomore charts a volatile and highly engaged relationship between audience, practitioner and critic (Bottomore 2001, pp. 129–130). Around 1907 these journals not only urged a business strategy which maximised the attraction of sound effects but began to argue the use of sound to clarify plot. Some cinema managers, sensing an appetite in their audiences, stepped up to a fervour the sounding of film in their establishments (Bottomore 2001, p. 133). Not surprisingly, this fervour brought itself undone. By 1908, film presentations were so noisy there was a call for sound to have a “psychological bearing” upon the scenario (Bottomore 2001, pp. 135-136).

41 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. By 1909, the ubiquity of sound effects, previously hailed as an antidote to the silence of moving pictures, was coming to be regarded as a threat to understanding those pictures. The “silence of death” in films was to be avoided but so was “cacaphonic embroidery” (Bottomore 2001, p. 132). By 1910, a backlash arose and industry magazines began to write of grievous misapplications of sound effects, and some critics called for the abolition of sound effects altogether. The criticisms against sound effects are instructive because they point to transgressions against (hitherto implicit) rules (in part aesthetic, in part semantic) governing the relationship between sound and moving pictures, and therefore serve as a source from which to extrapolate an inchoate theory of film-sound. Drawing upon Bottomore (Bottomore 2001) these ‘misapplications’ may be summarised as synchronisation, textural accuracy, loudness, too much sound, mood, literalism, genre, and situational relevance.

Bottomore’s taxonomy shows that theories and practices of sound design were embedded early in cinematic production and exhibition, understood positively as affordance and negatively as over-worked misapplications. Either way, it is clear that sound was being designed, composed and delivered throughout all phases of the history of silent cinema. From the mid-1910s through to the arrival of fully synchronous sound-film (1926–1931)68 the maturation of sound effects became critically recognised as a genuine cinematic element, albeit an element that must be used “judiciously” (Bottomore 2001, p. 135). Thus, leading up to the transition to synchronous sound-film, sound effects for moving pictures had evolved from a loose to a fully integrated audiovisual aesthetic practice.

2.3.3 Classical film-sound theory (1917–1960) Circumscribed between 1917 and 1960, classical cinema brackets the dynamism of an era that commences with the rise of the studio production model and ends with the dissolution of the vertically integrated Hollywood studio system. The historiography of classical film theory typically features the work of Sergei Eisenstein, Vsevolod Pudovkin, Rudolf Arnheim, Siegfried Kracauer, Béla Balázs and André Bazin. Although differing in their approaches, these theorists concurred in their argument that cinema was an art, and, like Münsterberg, sought to bring theories of medium specificity to bear upon their subject. However, just as early cinema revealed itself to be composed of multiple smaller movements, classical theories of film-sound may be parsed into early, mid and late phases. Eisenstein and Pudovkin exemplify an early-classical phase of theorising film while Arnheim, Kracauer, Balázs, and Bazin exemplify a mid-classical, and a late-classical phase is typified by such theorists as

42 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Elisabeth Weis, John Belton, Alan Williams, James Lastra, Amy Lawrence, Kaja Silverman Mary-Ann Doane, Rick Altman, and Michel Chion. While the early and mid phases can be understood as a refinement and buttressing of the theoretical position which considered sound ancillary to the visual image, the late-classical phase more fully explored sound as a productive partner in the cinematic experience.

Throughout the 1920s and early 1930s, rapid technological developments in the interrelated practices of commercial film production, exhibition and reception all occurred simultaneously in parallel markets around the world. As is well known, these developments resulted in both the normalisation of certain narrative conventions and in the application of particular stylistics for synchronised sound in the service of narrative advancement.69 This story of the investiture of ‘classical narrative’ cinema is much more widely told and understood than the story of the protean nature of sound in the era of purportedly silent cinema. In contrast to the previous section on early cinema, the classical period does not need to be so intensively revisionist. A quicker sketch of the classical era can deliver the necessary context, although, of course, there will be the need to highlight some less canonical figures and ideas that will bear upon the arguments concerning sound design as we move through the later chapters of this thesis.

The main impetus in the evolution of the classical period of commercial cinema can be characterised as the dialectic between formalism (also referred to as ‘illusionism’) and realism. This dialectic, simplified here for the sake of contextualising the role of sound, plays out as a tension between the drive to use cinema as a means of modelling possible or fantastical worlds (as notably advocated in the writings of Eisenstein, Arnheim and Balázs) versus the more realist drive to use cinema as a mode of observation of the characteristics of the given world (as best represented during the 1920s and 1930s in the corpus of Siegfried Kracauer right up to the English-language publication of his Theory of film: the redemption of physical reality (1960) and in the influential later writings of Andre Bazin such as his multivolume What is cinema? (Bazin 1967)). In the American industrial-cinema context in particular, as the preference for a realist aesthetic took hold, the regulatory influence of pictorial continuity editing and sustained, cohesive dialogue tended to persuade theorists of the role of the soundtrack was that of servant to the visual design and narrative drive of the realist scenography of the portrayed drama.

43 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. There were, of course, variants and resistances to this orthodoxy, both within American circles of production and criticism but also around the world. The robust debates and practical experiments amongst Russian formalist filmmakers in the early classical period are perhaps the most passionate of these intellectual tumults. Eisenstein, Pudovkin and Aleksandrov (1928), for example, foresaw how synchronous sound film threatened to tame cinema into a bourgeois form of ‘filmed theatre’ and sought to limit the ideological effects of dialogue by developing the concept of counterpoint and prescribing a contrapuntal audio-visual montage (Eisenstein, Pudovkin & Aleksandrov 1928). This is not to say the aural could not advance their revolutionary aims, only that sound and picture must be carefully combined. Pudovkin, in his ‘Asynchronism as a principle of film sound’ (1985 [1929]), develops a remarkably prescient blueprint for cinematic sound design that illustrates how audiovisual interaction can produce an as-if stream-of-consciousness to guide audience perception and comprehension. In a vein similar to Eisenstein, René Clair (Clair 1985 [1929]) was moved to specify overt rules to limit and control the use of sound. In these understandings of cinema, sound is characterised as a latecomer and an interloper, and a potential destroyer of the purity of a nascent visual artform70.

Conventional historical accounts of sound and cinema tend to concentrate upon the impact of the ‘talkie’ and the tumultuous years of transition to synchronous sound-film (Crafton 1997). The Jazz Singer (Crosland 1927) looms large in these historiographies, inaugurating, it was thought, “the beginning of the end rather than the end of the beginning” of cinema (Sarris 1998, p. 31). The figure of Dziga Vertov, however, presents a quite different account of sound and cinema. Vertov (pseudonym of David Kaufman) is another vibrantly skilled practitioner and formalist theorist from the early classical period. Entering medical training at the Psychoneurological Institute in Petrograd in 1914, the young Vertov was captivated by sound’s psychophysiological power and established a self-styled ‘Laboratory of Hearing’ to explore the possibilities of a sound art. Vertov’s sonic experiments parallel – and were informed by – Russolo’s ‘art of noises’ (Bulgakowa 2006). Futurist Luigi Russolo issued a challenge to the artistic community of his day to find art in noise. The music he was hearing in the concert halls seemed incapable of expressing the feeling of what it was to live in a rapidly transforming world. In The art of noises (1913), Russolo aches for a music that can express life in the modern world of steel and glass, of speed and electricity. He cast the music of the establishment as a pallid rendering of what might otherwise be, and dared those who might listen to break from “this restrictive circle of pure sounds and conquer the infinite

44 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. variety of noise-sounds”71 (Russolo 1913, p. 6). Ultimately, however, the limitations of the audio technology of the time led Vertov to abandon his experiments in sound collage and apply his ideas to moving pictures (Bulgakowa 2006; Fischer, L 1977/1978; Kahn 1999, pp. 139-144). Man with a Movie Camera (Vertov 1929) and Enthusiasm: The Symphony of Donbass (Vertov 1930), then, can be considered a visualisation of his sound experimental aims. Vertov’s avant-garde experiments and his concept of a radio-eye (Vertov 1984) presage ideas that were to be actualised in successive audio technologies and techniques that afforded higher fidelity recording and greater sophistication in multitrack editing and mixing, and so his line of praxis and inquiry represents an unfulfilled developmental opportunity for cinema even though the fecundity of these ideas remained latent within classical narrative realism.

After the early-classical period, film production and film theory matured into a mid-classical phase. Powerfully influenced by Kracauer’s later work on the realist aesthetic and with the widespread enthusiasm for Andre Bazin’s writings, theorising cinema fell to a new generation of theorists to complicate the realist normativity. The list of key theorists in this phase is very long indeed, but a few crucial figures should be identified as Christian Metz, Peter Wollen, Laura Mulvey, Kaja Silverman, Jean-Louis Comolli, Jean-Louis Baudry and Mary-Ann Doane. Once again, this is a rich and complicated field that deserves detailed and contentious accounting, but for the purposes of context one can describe this period as subject to the rise of structuralist, post-structuralist, psychoanalytic, and apparatus theory approaches, all connected to some degree by materialist concerns which focused on the effects of the cinematic system of vision and sound impacting the audience. Thus, the scene was set for the re-emergence of sound as a more explicit factor to be assayed in the production of meaningful cinematic experience, and it is in this phase that Sound Studies arises. For the purposes of this literature review, Rick Altman and Michel Chion, can serve as the orienting figures.

Altman and Chion: key theorists of film-sound At the beginning of this chapter it was asked if film-sound should be studied first and foremost as sound or in conjunction with the moving picture. In the work of Altman and Chion we have exemplars of both approaches.

Rick Altman is one of the most significant voices in the wave of Anglophone film-sound scholarship that arose in the last decades of the 20th century. His groundbreaking Cinema/Sound issue in Yale French Studies (Altman 1980b) appeared with the consolidation

45 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. of New Hollywood and announced the emerging field of Sound Studies72. Considered the “godfather of sound studies in the United States” (Hilmes 2005, p. 250), Altman is a fountainhead for critical approaches to film-sound (Altman 1992e, 1994; Altman, Jones & Tatroe 2000), film genre (Altman 1987, 1999b), and, as we have seen above, a revisionist understanding of the sound of early cinema (Abel & Altman 2001; Altman 2004).

In establishing and defending the legitimacy of a critical space within which to study film- sound, Altman identified “four and a half film fallacies” (Altman 1980a, 1992d) that disenfranchise sound within Film Studies. On Altman’s account these historical, ontological, reproductive, nominalist and indexical fallacies have impeded productive theory-building.

Briefly, the historical fallacy is the claim that “[c]inema was cinema before the sound track was added” and so sound cannot be fundamental to the cinematic experience (Altman 1992d, p. 35). The ontological fallacy, like the historical fallacy, refers to the misapprehension that “the image without sound still constitutes cinema, while sound without an image is no longer cinema” and “sound has (or should have) little effect on overall film structure” (Altman 1992d, p. 37). The corollary of these two assumptions is the apparent authority of the maxim cinema is a visual medium. Altman’s own research (Abel & Altman 2001; Altman 2004) has done much to reveal how these claims ignore the historical record. The reproductive fallacy refers to the misapprehension that audio technology merely ‘reproduces’ rather than represents sound while the nominalist fallacy refers to the confounding of idealised categories. Altman’s ‘half-fallacy’ is indexicality. Indexicality refers to the privileging of the concept of ‘recording’ as central to a definition of cinema, a concept Altman argues is outmoded. Even before the seismic shift in contemporary cinematic production from analogue to digital image-making, Altman understood that sound has “rarely been the result of straightforward indexical recording” (Altman 1992d, p. 43).

When articulating the ontological fallacy (Altman 1992d, pp. 37–39) Altman identifies an underlying ocularcentrism entangled in the assumptions of orthodox classical film theory. Rudolf Arnheim and Béla Balázs are obvious early proponents of the fallacy but moreover and “[s]urprisingly, this dependency on ontological arguments comes not from the enemies of sound, but from its greatest defenders” (Altman 1992d, p. 38). Altman identifies the work of Michel Chion, Mary Ann Doane, and Kaja Silverman as complicit in propagating the fallacy and is especially reproving of a prescriptive application of psychoanalytic theory over the

46 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. analysis of specific film soundtracks (Altman 1992d, pp. 38–39). Altman’s critique of the ontological fallacy has been reiterated and amplified by subsequent sound theorists, for example Jonathan Sterne (Sterne 2003, p. 142), Gianluca Sergi (Sergi 2004, p. 65-67), and Frances Dyson (Dyson 2009, p. 74).

In addition to the above fallacies, Altman (Altman 1985, 1992b, 1992a) has also identified the issue of a lack of explanatory power in the terminology available for film soundtrack analysis. This is a recurring theme within film-sound discourse. Jonathan Rosenbaum (1978), Norman King (1984), and William Johnson (Cumbow & Johnson 1985; Johnson, W 1985, 1989) have each critiqued the persistent and chronic problem of a deficient critical vocabulary. To countermand this, Altman argues for an understanding of cinema not as text but as event, and such an approach requires a unique and specific terminology. Eschewing musical terminology, Altman opts for terms such as spatial signature, auditory perspective and point- of-audition, reflection, echo and reverberation, sound hermeneutic, sound scale, and the cocktail party effect (Altman 1992b, 1992c). These terms reference the materiality and felt qualities of aural objects and events, subjective acoustic space, and auditory attention, and as such this terminology shares a critical perspective with the field of ecological psychoacoustics. Notably, Altman wishes to “restore a sense of sound’s role in creating our sense of the body” (Altman 1992d, p. 39).

Altman is particularly reproving of what he considers inappropriate conceptual models imported from external domains. Music is one such domain. Altman argues music is not metonymic for the soundtrack nor is musical terminology exhaustive of film-sound for “[w]hile all film sounds have loudness, pitch, and timbre, not a single sound in cinema can be adequately described with this musical terminology” (Altman 1992a, p. 16). The kind of real world complexity Altman seeks becomes apparent when he observes

[t]he middle C located on the first line below the G clef may be only a concept, but the sound that we hear with our ears – whether on the street or in a movie theater – is a heterogeneous event that carries its own temporal and spatial dimensions and constitutes a full-fledged narrative. When we listen to recorded sound we are therefore always listening to a particular account of a specific event. (Altman 1992a, p. 16)

47 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Michel Chion is arguably the most influential figure on the issue of film-sound terminology. Chion is a composer of musique concrète, a former critic for Cahiers du cinéma and one of the most heavily cited theorists of film-sound. His early studies in music and literature led him to work under Pierre Schaeffer at ORTF, the French Radio and Television Organization. Exposed to Schaeffer’s practice of musique concrète, Chion began to develop his own approach to conceptualising, assembling and mixing sounds (Anon. 2016; Fairfax 2017). If Altman’s ‘Cinema/Sound’ issue of Yale French Studies (Altman 1980b) opened the territory for theorising of film-sound Chion’s work took root in that tilled soil. Published in French in the 1980s (La voix au cinéma (Chion 1982) and Le son au cinéma (Chion 1985)), Chion’s ideas about film-sound appeared in the Anglophone world during the mid- to late 1990s as Audio-vision: sound on screen (Chion 1994) and The voice in cinema (Chion 1999). Since the publication of Audio-vision, Chion has reiterated his thinking on sound with Film, a sound art (Chion 2009) and, most recently, Sound: an acoulogical treatise (Chion 2016). Chion’s contribution to the study of film-sound is ample, and his work provides both a foundation and a point of departure for this thesis.

Confronted with the conceptual vacuum surrounding film-sound, Chion sought to ground his thinking by turning to the pathfinding work of his mentor, Pierre Schaeffer. Schaeffer sought to rebuild music theory by moving in the opposite direction to traditional approaches to music composition73. Instead of starting with the abstraction of musical notation and making it audible, Schaeffer took audible sounds and from it abstracted musical structure. Schaeffer’s semiotic approach to the musical organisation of everyday sounds culminated in musique concrète (Schaeffer 1952).

Listening modes Chion’s intellectual debt to Schaeffer is manifest in the honing of Schaeffer’s concept of listening modes to the filmic situation. Where Schaeffer (Schaeffer 1966) proposed four modes of listening Chion posited only three – causal, reduced and semantic listening (Chion 1994, pp. 25–34) – and these modes have become the dominant taxonomy within film-sound theory. Briefly, causal listening refers to a focus on the source of a sound. That is, attention is placed upon the actions and events that cause the sound rather than the sound per se. Reduced listening refers to listening to the material qualities of the sound itself, and is divorced from cause, context and semantic meaning. Reduced listening is also known as acousmatic listening because the sound source is hidden from the listener74. Semantic listening refers to

48 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. learned and culturally determined codes used to interpret a message. Chion’s modes are almost identical to those of Schaeffer except that Chion excludes Schaeffer’s concept of ouïr (perceiving). According to Schaeffer, ouïr is the most primordial of his four modes and refers to sound transduced and processed by the auditory system (Caramiaux et al. 2014, p. 36). Thus, problematically, Chion’s taxonomy severs and excludes auditory perception at its most basic and biological level75.

Chion’s terminology Chion’s most pronounced contribution to film-sound theory is the identification and labelling of a vast array of audio-visual relations and phenomena. Like Altman, Chion recognised “everyday language as well as specialized musical terminology is totally inadequate to describe the sonic traits that are revealed when we practice reduced listening on recorded sounds” (Chion 1994, p. 31). Chion generated many, many new terms in response to this terminological vacuum. Indeed, Chion’s apparently inexhaustible capacity to classify has earned him the moniker “the Linnaeus of film sound” (Corne 2016, p. 229). Chion’s nomenclature, first appearing in English in Audio-vision: sound on screen (Chion 1994, pp. 221–224) and expanded in Film, a sound art (Chion 2009, pp. 465–500), goes a long way toward filling the vacuum he first encountered, and is arguably the most referenced lexicon for sound among film scholars76. Audio-vision, synchresis, added value, acousmêtre, audiovisual contract, audiovisual scene, rendering and superfield are some of the most commonly encountered of Chion’s terms in film-sound discourse.

Audio-vision is the central concept around which all else orbits in Chion’s work. Indeed, Chion’s stated objective is “to demonstrate the reality of audiovisual combination – that one perception influences the other and transforms it” (Chion 1994, p. xxvi). With the concept of audio-vision Chion identifies and champions the inherently audiovisual nature of cinema, and prepares the ground for an aesthetics of multimodal interaction because audio-vision is the

type of perception proper to the experience of film and television, but often in real life as well, wherein the image is the conscious focus of attention but to which the sound supplies at every moment a series of effects, sensations, and meanings that often, by means of a projective phenomenon I call added value, are credited to the image and seem to emanate naturally from it. (Chion 2009, p. 469, emphasis in original)

49 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Synchresis is Chion’s portmanteau (synchronism plus synthesis), and is defined as the “spontaneous and irresistible weld produced between a particular auditory phenomenon and visual phenomenon when they occur at the same time” (Chion 1994, p. 63). For Chion, synchresis is innate but it is also influenced by culture (Chion 1994, pp. 63–64). Added value describes “the sensory, informative, semantic, narrative, structural, or expressive value that a sound heard in a scene leads us to project onto the image, creating the impression that we see in the image what we are in fact audio-viewing” (Chion 2009, p. 212).

Filmic aural and visual phenomena interact through what Chion calls an audiovisual contract. The audiovisual contract is “a sort of symbolic pact to which the audio-spectator agrees to forget that sound is coming from loudspeakers and picture from the screen” (Chion 1994, p. 222). Formulated in this way the audiovisual contract is akin to a ‘willing suspension of disbelief’, or (possibly more accurately) a form of transportation or immersion. However, Chion claims this ‘contract’ is “the opposite of a natural relationship arising from some sort of preexisting harmony among the perceptions” (Chion 1994, p. xxvi) suggesting a personal or social agency is at work in determining those sensory interactions. This line of argument has resonated greatly within Film Studies. Beck and Grajeda, for instance, take the audiovisual contract as the organising idea for their edited volume Lowering the boom: critical studies in film sound (Beck & Grajeda 2008a, p. 3) and, off the back of Chion, regard “[s]ensory perception as culturally constructed” (Beck & Grajeda 2008a, p. 3).

Chion has attained prominence in film-sound theory for his identification of audiovisual phenomena but he has also sparked contention with his concept of the acousmatic voice – the acousmêtre – and his claim “there is no soundtrack”.

In Audio-vision (Chion 1994) and The voice in cinema (Chion 1999), Chion advances the idea that cinema is vococentric. That is, cinema “almost always privileges the voice, highlighting and setting the latter off from other sounds” (Chion 1994,p. 5). Vococentrism is “the process through which the voice spontaneously attracts and centers our attention in a mixture of sounds” (Chion 1994, p. 5). Chion elevates the importance of the voice because it is the conduit for language. Coupled with the concept of acousmatic sound, Chion coined the term acousmêtre (Chion 1994, pp. 129–131). The acousmêtre is a disembodied voice which possesses a mystery and a power drawn from its omniscience because it exists neither inside nor outside the visual frame. The acousmatic voice is the voice of Samantha in Spike Jonze’

50 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Her (Jonze 2013) and Hal in Kubrick’s 2001: A space Odyssey (Kubrick 1968), the wizard in Fleming’s The Wizard of Oz (Fleming 1939), and, of course, the mother in Hitchcock’s Psycho (Hitchcock 1960). An important aspect of the concept is not only the power the acousmêtre appears to possess but its astonishing loss of power once it is inscribed inside the visual field.

Chion’s ideas concerning cinematic voice, particularly the acousmêtre, owe a debt to psychoanalytic film theory. Chion situates his concept of the acousmêtre in Lacanian terms, and although this places him in the general current of theory dominant at the time of his writing it has attracted critique. Feminist film theorists, in particular Kaja Silverman (1988, pp. 74–77), have judged Chion’s writings on the acousmêtre harshly for its propagation of traditional notions of gendered voice. Silverman condemns Chion’s approach to sexual difference that categorises a ‘masculine’ and ‘feminine’ filmic voice as “motivated primarily by the search for poetic props, and so remain for the most part both uncritical and devoid of self-consciousness” (Silverman 1988, p. 49).

When discussing his concept of the audiovisual scene, Chion is sceptical of the possibility of an autonomous and integrated soundtrack, claiming:

If we can speak of an audiovisual scene, it is because the scenic space has boundaries, it is structured by the edges of the visual frame. Film sound is that which is contained or not contained in an image; there is no place of the sounds, no auditory scene already preexisting in the soundtrack—and therefore, properly speaking, there is no soundtrack. (Chion 1994, p. 68).

Asserting that sound has neither coherence nor autonomy in relation to the visual image and is instead automatically subsumed into the visual narrative, Chion also seems to assume character and action are properties that somehow belong to vision but not to sound, and so reinstates the classical hierarchy where vision is ‘king of the senses’. Chion’s vertical schema is telling and hints at an underlying ocularcentrism. Moreover, the claim “there is no soundtrack” exhibits a paradox in Chion’s thinking. While championing the concept of audio- vision and the recognition of cinema as an audiovisual medium, Chion resists the possibility that visual and aural imagery are commensurable. Chion appears to suggest vision and audition operate via different perceptual rules: a dubious assumption which requires

51 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. validation. Chion also appears to conflate visual imagery with mental imagery to the exclusion of aural imagery. These are assumptions with far reaching implications yet Chion marshals insufficient collaborating evidence. Central to Chion’s argument is the assertion that sound lacks a counterpart to the visual frame/shot. For him, the frame is a container within which visual elements are ordered – the mise en scène – and because Chion believes film- sound possesses no equivalent unit he takes the visual shot/frame-as-container to be cinema’s primary organising principle. But is this true, and how may we test Chion’s assertion that sound-sound interaction is irrelevant or nonexistent?

Altman and colleagues McGraw Jones and Sonia Tatroe (2000) find Chion’s argument unpersuasive. For them “film sound – taken as a single complex unit rather than as three or more separate components – cannot be understood without analyzing relationships among soundtrack components” (Altman, Jones & Tatroe 2000,p. 341). Recognising the utility of the concept of mise-en-scène, Altman et al. have developed its aural counterpart, the mise-en- bande. Where “mise-en-scène analysis foregrounds relationships among image components; mise-en-bande analysis concentrates on the interaction among the various components making up the soundtrack” (Altman, Jones & Tatroe 2000,p. 341). The mise-en-bande sets aside Chion’s assumptions because it prioritises the analysis of “intercomponent, intrasoundtrack relationships” (Altman, Jones & Tatroe 2000,p. 341) and in the process reveals interacting sonic planes of dialogue, music and effects (DME). The resulting multiplane sound environment refutes the idea that sound-sound interaction lacks coherence or is secondary to interaction with the visual image77.

Relevant to a richer understanding of the mise-en-bande is research which reveals how the listener automatically groups sonic information to form sound objects. Psychologist Albert Bregman’s concepts of auditory scene analysis and auditory streaming (Bregman 1990, 2002, 2008) are of particular significance in this discussion. Auditory scene analysis refers to the “mental representations of individual sounds from the summed waveform that reaches the ear of the listener” (Bregman 2008, p. 861). Auditory streaming is a

perceptual interpretation, produced by auditory scene analysis … heard as separate coherent patterns evolving over time (auditory streams), and interpreted as individual acoustic sources present in the auditory environment (such as a voice, a violin, or a machine) … A stream (in ordinary conversation, we call it a sound or a sequence of

52 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. sounds) is a perceptual entity, analogous to a visual object, capable of binding a cluster of properties (e.g., one sound is high in pitch, loud, and on the left, whereas the other sound is low in pitch, soft and intermittent, and straight ahead). (Bregman 2008, p. 861).

A feature of auditory scene analysis (ASA) is that it proceeds through the binding of acoustic qualities which are temporally coherent. Broadly, the principle is that a sound’s components tend to be emitted simultaneously from the same spatial location thus lending the auditory system a heuristic by which to parse the environment. Such a heuristic not only also furnishes a sound’s perceived spatial position but also its pitch, timbre and loudness. Chion’s claim “there is no soundtrack” appears rooted in a privileging of the spatiality of the visual image. Cinema, however, is a spatiotemporal system which engages both audition and vision. Bregman’s ASA model offers a way to reclaim the temporal dimension of the cinematic image which is missing from Chion’s analysis. Applying Bregman’s concepts to cinematic sound, the listener can be said to perceive the soundtrack as a meta-sound object comprised of auditory streams which interact to form the perception of events. It is the role and function of sound design to operationalise the elements of the soundtrack – dialogue, music and sound effects (which are themselves comprised of sound fragments) – in accordance with these innate perceptual rules78.

Bregman’s research provides empirical support for the claim by Altman and his colleagues that “[f]or a soundtrack to exist as an independent entity, individual sounds have to be constructed in relation to each other as well as to the image” (Altman, Jones & Tatroe 2000,p. 356). While Altman’s multiplane concept of the soundtrack examines sonic interaction at the macro level of the DME, Bregman’s auditory scene analysis extends to the micro level of sensory qualities which bind together to form an auditory stream all of which happens automatically and before crossmodal fusion with vision. In light of Bregman’s research, Chion’s assertion "there is no auditory container for film sounds” and that sound-sound interaction is of no consequence fails to ring true.

Another critic of Chion’s claim is William Whittington. Whittington considers the technical design of auditoria and microphones to produce instances of an ‘acoustic container’, arguing

53 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. the theatrical environment itself forms a frame in which specific fields of sound are deployed, and that microphones do have frames in terms of their pickup patterns that limit the angle and range of recording, which in turn provide spatial cues that allow us to form auditory boundaries that are sometimes more powerful than those presented by the image. (Whittington 2010, p. 258)

Whittington’s idea of “auditory boundaries” is salient because it draws attention to the perceptual mechanisms that underpin the formation of sound objects. Although he does not refer to Bregman’s work, Whittington’s observations are readily grounded and enhanced by the concepts of auditory scene analysis and auditory streaming. Whittington’s observations also point to implicit knowledge used in professional practice. Not only are microphones designed with differing sensitivities and affordances (for example, dynamic versus condenser types, and cardioid, hypercardioid, and omni-directional pick-up patterns), but these acoustic affordances are specifically used by sound recordists at the site of production to fashion sound objects, and are further manipulated and enhanced through the subsequent post-production phases of editing and mixing. Lastly, it is interesting to note, in the contemporary era of surround sound, Chion’s claim “there is no soundtrack” is of far less consequence to Whittington than the more pressing question of cinematic immersion. (Whittington 2010)

Chion’s controlling idea that sound is exclusively vertically integrated into visual narrative can be historically situated as part of the 20th century’s linguistic and visual turns. In some ways, Chion’s extended focus on cinema as a ‘verbocentric’ art fits this profile, but there is also the underlying issue of visual narrativism. The corollary of visual narrativism is, of course, aural narrativism, but to date such a field is ill-defined and under-theorised. In their prolegomena to Audionarratology (Mildorf & Kinzel 2016b), Jarmila Mildorf and Till Kinzel observe “aural media … have not necessarily been investigated with a view to identifying their specifically narrative qualities, affordances and potentialities” and “as far as narratology is concerned, sound in many art forms and medial contexts seems to have been relegated to what Leibniz called petites perceptions” (Mildorf & Kinzel 2016a, pp. 2–3) where petites perceptions are sense perceptions below the threshold of awareness. If a narratology of sound can be said to exist it tends to focus on human language. To illustrate the paucity, Mildorf & Kinzel note not a single entry for ‘narrative’ or ‘story’ exists in Jonathan Sterne’s influential The audible past: cultural origins of sound reproduction (Sterne 2003), and suggest an “acoustic turn” in narratology is essential. Ultimately, the nature and function of aural

54 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. narrative is under-explored, and this lack leaves Chion’s assertions awaiting clarification and verification.

Another limitation to Chion’s oeuvre is a dawning sense of anachronism. Whittington (2010, p. 256) has drawn attention to the fact Chion appears to have developed his ideas about film- sound based upon the audio-viewing of VHS tapes. As Whittington points out, VHS is a Dolby Stereo encoded format and cannot reproduce a modern movie’s multichannel experience of sound space. Given Dolby Stereo emerged in the mid-1970s and was superceded by various digital surround sound formats during the early 1990s, it appears the ramifications of digital sound are all but missing from Chion’s thinking.79 Whittington also identifies a number of gaps and confusions in Chion’s understanding of cinema sound technology, such as his lack of distinction between Dolby Stereo and other Dolby technologies, and a conflation of multitrack sound design techniques and multichannel sound reproduction under the rubric of ‘multitrack sound’ (2010, pp. 256–257).

These confusions and conflations raise questions concerning the contemporary utility of Chion’s ideas, especially with regard to his ideas of superfield and acousmêtre. Within digital surround sound cinema, Chion’s superfield is sublimated and the acousmatic voice ceases to function within the multichannel environment of what Mark Kerins terms the ultrafield (Kerins 2010, p. 92). Chion acknowledges as much.

When people ask me what happens to the acousmêtre with 360-degree cinema, or virtual reality, I say: “It’s different. I have no idea.” Personally, I am less stimulated by omnidirectional forms of expression. It’s not an area I’m competent in. I don’t think it’s bad, but there is no more acousmêtre. The acousmêtre no longer functions. (Fairfax 2017)

Chion’s admission is troubling for several reasons. Firstly, it suggests his hallmark concepts of superfield and acousmêtre are technologically delimited. Secondly, Chion displays little interest in hypothesising the underlying or enduring sonic principles of audiovisual media in toto. His interest lies in observing the effects of film-sound as it manifested under certain technological and historical conditions of the late 20th century. This sense of anachronism only deepens when confronted with surround sound deployed in audiovisual media such as computergames and VR. Paradoxically, Chion appears to resist the concept of immersion

55 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. even as he acknowledges the rendering of sensation is of great filmic importance (Chion 2013, p. 327). For Chion, it may be that forms like computergames and VR appear too dissimilar from film because “[i]nteractivity alters our perception” (Chion 2013, p. 329), but if so it is not clear why this should bear negatively upon theories of audio-vision80. After all, cinema is also a form of virtual reality albeit one that precludes the capacity for in-world interaction.

The enduring legacy of Chion’s theorising of film-sound is his central concept of audio-vision and the framing of cinema as an audiovisual medium. His specialised terminology identifies and labels previously uncatalogued audiovisual phenomena and this lays the foundation for an audiovisual aesthetics. However, some of his neologisms are conceptual ‘black boxes’ that conceal rather than explain the underlying principles of cinematic sound design. Another weakness in Chion’s approach is its conceptual insularity and a requirement for his more contentious claims to be validated.

Another aspect to Chion’s legacy is his indebtedness to Schaeffer and the application and dissemination of his mentor’s ideas into the field of Film Studies. However, Chion is not unique in drawing upon Schaeffer’s foundational work. R. Murray Schafer (1973, 1977, 1993), a music composer and educator, had previously drawn upon Schaeffer in establishing the study of environmental soundscapes and a nascent acoustic ecology with Barry Truax (1978, 2008), Hildegard Westerkamp (Jordan 2007; Kolber 2002; Norman 2004) and others from Simon Fraser University. Similarly, Barbara Flückiger (2007), in formulating a taxonomy for the description of filmic sound effects, draws upon Chion and hence upon Schaeffer. Schaeffer’s pioneering work is undoubtedly a touchstone for much sound theory and practice81, and the prominence his body of thought enjoys may result from it being the first to attempt to operate across the tripartite of ‘noise’, speech and music. However, the intellectual legacy which connects Schaeffer to Chion, Schafer and Flückiger dominates and delimits film-sound theory to assumptions implicit within electroacoustic music theory82. This alone is cause for concern. Musique concrète subordinates sound to an already idealised musical structure. Such a process captures only some aspects and properties of sound but leaves others as an excess to the needs of musical structure, and are either eviscerated or suppressed. This is manifest in Chion’s bold claim “I think you have to understand music to be able to talk about the use of sound in cinema” (Chion quoted in Warburton 2008). Implicitly, and troublingly, the concepts of musical abstraction which inform electroacoustic

56 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. music theory are assumed to be applicable to, and exhaustive of, an understanding of cinematic sound even though there is growing evidence to suggest music (and also language) evolved from “a common emotional signal system based on the imitation and modification of environmental sounds” (Ma & Thompson 2015).

Chion’s work in locating cinematic meaning in the processes of multimodality provides a background for this thesis. When few conceptual tools were available for the analysis of film- sound Chion’s neologisms provided a powerful catalyst to a burgeoning field of study. However, these same terms have come to present friction to interdisciplinary engagement. Moreover, Chion’s prismatic terminology does not form an integrated account of cinematic sound. This recalls both Jonathan Sterne’s lament concerning Sound Studies generally, that there exists “a vast literature on the history and philosophy of sound; yet it remains conceptually fragmented” (Sterne 2003, p. 4), and Edward Slingerland’s observation that “too much of current work in the humanities resembles butterfly collecting … a lack of any sort of guiding theoretical framework to help researchers formulate productive research questions and to make sense of their data” (Slingerland & Collard 2011b, p. 33). The point of departure of this thesis takes the form of bringing to bear an interdisciplinary approach to cinematic sound to verify and expand upon the legacy of Chion and Altman with information and concepts arising from the perceptual, affective and cognitive sciences. As the work of Chion and Altman gestures toward, Film Studies can gain much from a ‘third culture’ understanding of the psychophysics of audiovisual phenomena that underpin the cognition of cinematic experience and narrative.

With the evolution of analogue classical cinema into the digital post-classical, “cinematic experience shifts from watching a movie to being immersed in the diegesis of one” (Kerins 2010, p. 141, emphasis in original), and this returns us to the need flagged by Whittington for a “broader conversation about sound and immersion in sound studies” (Whittington 2010, p. 258). Chion’s work halts at the threshold to an exploration of cinematic immersion and only glancingly approaches issues of affect (although his synaesthetic notion of the ‘trans-sensory’ (Chion 2013) suggests how his ideas might connect to broader studies of perception, affect and consciousness). Launching from this point, and of particular relevance to this thesis which seeks to explore the affective function of cinematic sound, is contemporary media psychological research which is uncovering the intimate entanglement of sound, affect, immersion, and presence (Baños et al. 2004).

Another significant threshold in Chion’s work and standard film theory more generally is the issue of visual narrativism. It is worthwhile noting at this point that while Chion is indebted to Schaeffer there is an important point of difference between the two thinkers. Schaeffer gave leading importance to “audiovisual texture, partially freed from the obligations imposed by film narration” while Chion weighted importance to narrative (Bizzaro 2011, p. 9). For Schaeffer, the modal qualities of an audiovisual work enter into a direct interaction “independently of the topic being narrated, the aesthetic message is conveyed entirely by the formal organization of the objects, visual images and sound modulations” (Bizzaro 2011, pp. 6-7). This suggests a form of meaning always arises independently of the determinations of narrative. In turn, this implies the operation of two distinct yet interacting sense-making processes. The first emerges from the evanescence of sensory perception while the second is a structuring of sensation through narrative logic. This dual-process structure, which I theorise in Chapter 5 as perceptual immersion and narrative immersion, is evident in Schaeffer’s claim that “[e]ncountering the concrete starting from the abstract, this is the great invention of language; encountering thought starting from things, this is the invention of radio and cinema” (Schaeffer quoted in Bizzaro 2011, p. 4).

2.3.4 Post-classical film-sound theory (1960–present) If the classical period may be characterised as an era of relative stability then the turbulence which developed throughout the 1960s signals a shift to a different kind of cinema, particularly in the United States83. Consequences of the US Federal anti-trust suit brought against Hollywood – the so-called 1948 Paramount Case – impacted the way movies were developed, produced and exhibited while the rise of television and the coming-of-age of the baby-boomer generation greatly altered traditional viewing habits and preferences. These factors led to a gradual demise of the classical filmmaking practices rooted in the studio system, and afforded the emergence of independently produced movies that became known as the ‘American New Wave’. This was a movement, largely counter-cultural, which appeared in the mid-1960s but which evolved by the mid-1970s into the ‘New Hollywood’ (Schatz 1993, 2009). The New Wave was markedly different in feel to classical studio fare. It portrayed darker themes, typically of urban landscapes suffering institutional or social moral decay. Arthur Penn’s Bonnie and Clyde (Penn 1967) is touted as the watershed, and Easy Rider (Hopper 1969), Midnight Cowboy (Schlesinger 1969), The French Connection (Friedkin 1971) and Mean Streets (Scorsese 1973) further exemplify the movement’s visceral

58 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. style. The polemics softened with the shift to New Hollywood but techniques to elicit corporeal responses in the audience intensified. Instead of an existential bleakness, movies such as Spielberg’s Jaws (Spielberg 1975) and Close Encounters of the Third Kind (Spielberg 1977) and Lucas’ Star Wars (Lucas 1977) presented spectacular scenarios eliciting excitement, fear, wonder and awe.

The heightened sensations of the New Wave and New Hollywood evince “the spectacle of a body caught in the grip of intense sensation” (Williams 1991, p. 4), qualities which Linda Williams has identified as intrinsic to the ‘body genres’ of horror, pornography and melodrama (Williams 1989, 1991). Williams’ concept of ‘body genres’ shifts emphasis away from the traditional film theoretical focus on character-based identification and toward the significance of the audience’s embodied responses to cinematic imagery. As the New Hollywood gathered momentum, the cinematic techniques for eliciting such responses massed into what Geoff King calls ‘impact-aesthetic’ (King, G 2000, 2003) or what David Bordwell labels ‘intensified continuity’ (Bordwell 2006). Steven Shaviro, however, considers the current era of cinema to have moved beyond Bordwell’s concept of New Hollywood intensification into a phase of post-continuity (Shaviro 2010), a phase also termed by Carol Vernallis as ‘cut-up cinema’ (Vernallis 2013) and Matthias Stork as ‘chaos cinema’84 (Stork 2013). Percolating within Film Studies at the close of the 20th and the beginning of the 21st centuries is a conception of the audience as affectively embodied.

The turn to affect and the corporeal The turn to affect and the corporeal occurred within the humanities and social sciences in the mid-1990s as a response to the perceived epistemic boundaries imposed by a privileging of semiotic textual analysis over lived experience. Patricia Clough observes the turn to be “the movement in critical theory from a psychoanalytically informed criticism of subject identity, representation, and trauma to an engagement with information and affect” (Clough 2007, p. 2). The shift in paradigm was toward the political and cultural dimensions of bodily experience, particularly through Deleuzian notions of assemblage and affect. Brian Massumi (Massumi 1995) and Eve Sedgwick (Sedgwick & Frank 1995, 2003), key theorists presaging the turn, employ conceptualisations of affect which trace through Deleuze and Guattari to Henri Bergson and Baruch Spinoza (and, in the case of Sedgwick, also Silvan Tomkins’ non- Freudian affect theory). A characteristic of the Deleuzian school is its conception of affectivity as “a substrate of potential bodily responses, often autonomic responses, which are

59 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. in-excess of consciousness”(Clough 2007, p. 2), a characteristic that draws into question the veracity of the conceptual dualities of body/mind and emotion/reason.

Affect and emotion, it is important to note, are not considered identical in these studies. Affect is characterised as nonconscious, ‘pre-cognitive’ and ‘pre-personal’ whereas emotion is conscious and subjective85. Affect is understood as “a nonlinear complexity out of which the narration of conscious states such as emotion are subtracted” (Clough 2007, p. 2), an observation which resonates with the embodied cognition approach I undertake in this thesis. Moreover, emotions are understood in these studies as forms of social and cultural practice – ideological states of the individual – that deepen “constructivist notions of power, performativity and activity … [and are] extended beyond the socio-cultural domain to include pre-conscious and pre-discursive forms of existence” (Greco & Stenner 2008). This opens the way for a culture-theoretical engagement with autonomic and nonconscious processes. On this account, the membrane between affect and emotion is thin and not explicitly unidirectional; emotion has the capacity to loop back to alter bodily affect (for example, through a medium’s aesthetic effects). By this line of argument, affect is not ‘pre-social’, and Sara Ahmed’s The cultural politics of emotion (Ahmed 2004) is a key text in framing bodies, emotion and language when developing an intersectional politics of sexuality, gender, race, and class. What is foregrounded is a purported emancipatory potential of affect. Additionally, the turn to affect also extends to the study of technologies that afford the production and exhibition of bodily responses, expanding purview beyond the human to reconfigure previous understandings of materiality, bodies (both human and non-human) and technology (Clough 2007, p. 2).

The turn’s focus upon materiality has significantly reconstrued the historically tense relationship between the social and natural sciences. However, while the affective turn in the social sciences may have drawn inspiration from fields such as quantum physics and neuroscience it should not be considered synonymous with the affective turn in the natural sciences. Clare Hemmings (Hemmings 2005), Constantina Papoulias and Felicity Callard (Papoulias & Callard 2010), and Ruth Leys (Leys 2011), for example, have identified a series of cross-disciplinary missteps in the cultural theory of Brian Massumi (Massumi 2002a), William Connolly (Connolly 2002) and Nigel Thrift (Thrift, NJ 2008) which have led to the misinterpretation and misappropriation of a range of neuroscientific concepts for the purposes of a distinctly political project. All this is to say the conceptualisation of affect, emotion,

60 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. feeling and mood is exceedingly messy, both in the social and the natural sciences, and the social constructionist theories of affect which dominate the humanities are but one of several approaches. Acknowledging this messiness, and because the nature and function of affect is at the heart of my proposed account of cinematic sound, I explore it in detail in Chapter 4 using the multilevel approach set out in Bartsch & Hübner’s emotional communication model (Bartsch 2004; Bartsch & Hübner 2005).

Film Studies and the body The turn to affect and the corporeal provides backdrop to recent critical approaches within Film Studies86. Vivian Sobchack (Sobchack 1992, 2004), Steven Shaviro (Shaviro 1993, 2010), Elena del Rìo (del Rìo 1996, 2008), Laura U. Marks (Marks, LU 2000, 2002), Barbara Kennedy (Kennedy 2000), Jennifer M. Barker (Barker 2009, 2016), Tarja Laine (Laine 2006, 2007, 2011), and Anne Rutherford (Rutherford 2003), among others, exemplify the rise of a “postmodern scopic regime”87. This regime relates to, but differs from, that of Linda Williams in that bodily sensation and affect are not a passive “index of response” but “active tools” for understanding cinematic imagery so that “the body is not only a feeling machine driven by visual images but an understanding and perceiving machine that draws on encysted memory, experience and knowledge, to provide a subcognitive ground for its own responses” (Elliott 2011, pp. 32–33). These theorists, leveraging Merleau-Ponty and Deleuze, conceptually cluster sensation, affect, embodiment, and multimodality (particularly the synaesthetic relationship between vision and the tactile) for the purposes of cultural critique. Shaviro, for example, is concerned with the interplay between affect entailed by post-classical cinema and “the delirium of globalized financial capitalism” (Shaviro 2016, p. 60). For Sobchack, human existence is radically material and embodied and it is this embodiment that affords empathy and ethics (Sobchack 2004, pp. 3 and 8) while for Marks the concepts of embodiment and affect afford exploration of the cinema of diasporic cultures and intercultural exchange, particularly through sensory memory.

Post-cinema, post-continuity, the cinematic body, and post-cinematic affect are central concepts in Shaviro’s thinking (Shaviro 1993, 2010). Referencing Raymond Williams, Shaviro posits post-cinematic affect as a unique ‘structure of feeling’ produced by post- cinema. With the transition from analogue to digital modes of production, hand-in-hand with neoliberal economic practices, Shaviro argues a new media regime has arisen to reshape the articulation of lived experience (Shaviro 2010, p. 2). Shaviro’s professed goal is to provide an

61 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. account of “what it feels like to live in the early twenty-first century” where audiovisual mediaworks are “machines for generating affect, and for capitalizing upon, or extracting value from, this affect” (Shaviro 2010, p. 2, emphasis in original). Post-continuity, according to Shaviro, is an audiovisual style that arises from post-cinema; it is “a preoccupation with immediate effects [that] trumps any concern for broader continuity – whether on the immediate shot-by-shot level, or on that of the overall narrative” (Shaviro 2010, p. 123). What is pertinent in Shaviro’s work to this thesis is the centrality of affect, and, moreover, the function of cinematic style in the generation of such affect, particularly affect that is not necessarily in the service of narrative.

Shaviro follows Deleuze when conceiving of human experience as fundamentally affective. However, while Deleuze urges it is “not to psycho-analysis or linguistics but to the biology of the brain that we should look for principles” of cinema (Deleuze quoted in Elliott 2011, p. 35), Shaviro does not appear to seek out neurobiological research directly. Shaviro’s critique does, however, identify issues with the cognitive film theory advanced by the decidedly ‘hard’ cognitivism of David Bordwell (Bordwell 1989c; Bordwell & Carroll 1996), although this is something of a strawman argument. Shaviro’s criticisms are correctly aimed at a classical form of cognitivism, that is, the first wave of cognitive science which Bordwell directly draws upon, but appears unaware of developments in the second and third waves which concentrate upon the body and affect, aspects of which are taken up by later cognitive film theorists such as Torben Grodal (Grodal 1997) and Greg M. Smith (Smith, GM 2003b). This blindspot is revealed in Shaviro’s own re-evaluation of his groundbreaking work when he acknowledges he had not yet read William James’s somatic theory of emotion at the time of writing The cinematic body (Shaviro 1993) even as he considers his form of affect/body theory to be stronger than the approach laid out by cognitive film theory (Shaviro 2008, p. 51). This gap reveals, firstly, the wicked problem involved in defining and explaining the complex relationship between affect and cognition, and, secondly, the dangers involved in developing an interdisciplinary approach to theorising cinema that is not directly and sufficiently engaged with all of its contributing domains. It is for this reason I provide an extension to this review of literature in Chapter 4 that examines the warp and weft of affective neuroscience and the cognitive sciences generally.

Vivian Sobchack’s theoretical approach complements and contrasts that of Shaviro. Adapting Merleau-Ponty in the service of developing a personal form of phenomenology, Sobchack

62 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. argues cinematic experience to be radically embodied and multimodal, asserting “film experience is meaningful … because of our bodies” (Sobchack 2004, p. 60, emphasis in original). Referencing the psychoneurology of Richard Cytowic, Sobchack identifies a radical multimodality as the source of a synaesthetic thickness that leads her to consider the viewer a “cinesthetic subject” (Sobchack 2004, p. 79). Sobchack develops a multimodal framework to understand cinematic imagery as synaesthetic imagery, drawing particular attention to “the embodied nature of vision, the body’s radical contribution to the constitution of the film experience” (Sobchack 1992, p. 25, emphasis in original). Drawing upon ideas from James J. Gibson’s theory of ecological perception, Sobchack resists the idea of a ‘pure vision’ to conceive of this multisensory engagement as an “embodied vision” (Sobchack 1992, p. 25). Such resistance is similarly echoed in other fields. Art historian W.J.T. Mitchell argues visual media are intrinsically multimodal media. Noting visualism has escalated to the level of fetish, Mitchell insists the term ‘visual media’ is “inexact and misleading … all the so-called visual media turn out to involve the other senses (especially touch and hearing). All media are, from the standpoint of sensory modality, ‘mixed media’” (Mitchell 2005, p. 257). And at the nexus of analytic philosophy and neuroscience a similar position is arrived at by Patricia Churchland, Vilayanur S. Ramachandran and Terrence Sejnowski (Churchland, Ramachandran & Sejnowski 1994) when they critique pure vision as a fiction, preferring the term ‘interactive vision’ in acknowledgment of the role of the sensorimotor system in a way that implicates the entire human body not just the five classical senses.

The power of Sobchack’s phenomenological approach is its placement of the lived body at the centre of human understanding because “[m]aking conscious sense from our carnal senses is something we do whether we are watching a film, moving about in our daily lives and complex worlds, or even thinking abstractly about the enigmas of moving images, cultural formations, and the meanings and values that inform our existence” (Sobchack 2004, p. 1). Sobchack’s insight resonates with the concept of embodied metaphor developed by Mark Johnson and George Lakoff (Johnson, M 1987; Lakoff & Johnson 1980a, 1999) which proposes conscious thought to be fundamentally metaphoric in nature, gaining meaning precisely because they emerge from bodily experience88. Within this framework, everyday and mediated experience become commensurable when the human body is acknowledged as common ground.

63 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. First-person phenomenological accounts of the kind offered by Sobchack provide important introspections about the feel of the world but can suffer limitations. Causal reports of bodily sensations and affects require a translation through consciousness and language, a process capable of cognising only a limited portion of our sensory experience (Kuipers 2008). Moreover, conscious explanations of one’s own responses, Nisbett and Wilson remind us, are sometimes wrong and that we are, in many ways, “strangers to ourselves” (Nisbett & Wilson 1977; Wilson, TD 2002). Beginning with the human level of the experience of cinematic images, phenomenological approaches drive upward to the cultural but forego a downward exploration of the neurobiology that is purported to underpin the phenomenological experience itself. An example of this is Sobchack’s use of neuroscientific research in crossmodality and synaesthesia that is only en passant. Phenomenological film theory, as Adriano D'Aloia observes, “theorises an embodied mind without any reference to the neural mechanism”(D’Aloia 2012, p. 221, emphasis in original). Between these two levels, between the neurobiological and the phenomenological, exists a deep rift. What is missing from both Deleuzian and phenomenological approaches to film theory is an explicit explanatory mechanism that roots consciousness in the processes of ‘the lived body’, jumping the Cartesian chasm to bind materiality and mind. The blend of the neuroscientific and the phenomenological in service of producing a ‘neurophenomenological’ method, such as proposed by Francisco Varela, Antoine Lutz and Evan Thompson (Lutz, A & Thompson 2003; Varela 1996), responds to the strengths and weaknesses inherent in both modes of investigation not as competing but complementary modes of knowing. In Film Studies, a neurophenomenological approach has been taken up by Michele Guerra and Vittorio Gallese (Gallese & Guerra 2012a), Adriano D'Aloia (D’Aloia 2012), Jane Stadler (Stadler 2016) and, I would argue, in my own line of research (Ward, MS 2015). Therefore, when developing a conceptual framework, it is important to recognise that neither the first-person of the phenomenological nor the third-person of the empirical can, on its own, provide a complete and satisfying explanation of cinematic experience and meaning. (More on this below when discussing Murray Smith’s notion of triangulation’ (Smith, M 2011, 2017).)

The corporeal turn in film theory has been heralded as a release from the dominance of vision. However, while the work of Sobchack, Marks, Barker and others strive away from an earlier ocularcentrism endemic in film theory, terms such as embodied vision (Sobchack) and haptic visuality (Marks) re-tool the trope of the cinema-eye by suggesting the value of synaesthetic thickness continues to be calculated in terms of an enriched vision. Even as the eye is

64 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. disavowed it is nonetheless reinstated as normative against which all else is marked. The sustained theoretical focus upon the synaesthetic thickening of vision while valorising a “cinesthetic subject” is somewhat perplexing given little of this work actively discusses the role of the aural, a modality that is obviously and directly addressed by the cinematic image. This is even more perplexing given sound literally touches the listener. Sound is felt on the skin and inside the body, not just the cochlea. This is not to say the aural fails to be considered in this work, but it does not seem to be of central concern. An exception to this tendency is the work of Lisa Coulthard (Coulthard 2012, 2013) and Martine Huvenne (Huvenne 2013) who have begun development of the concept of haptic aurality.

Where Marks deepens Sobchack’s cinesthetic subject to develop a cultural politics of the ‘skin’ and ‘body’ of the film, Jennifer Barker (Barker 2009, 2011) explores the potential of that body to elicit kinaesthetic responses. Elaborating a phenomenological approach based upon Merleau-Ponty, and building upon concepts developed by Sobchack and Marks, Barker explores notions of cinematic tactility. Cinematic tactility is “a general attitude toward the cinema that the human body enacts in particular ways: haptically, at the tender surface of the body; kinaesthetically and muscularly, in the middle dimension of muscles, tendons, and bones that reach toward and through cinematic space; and viscerally, in the murky recesses of the body, where heart, lungs, pulsing fluids, and firing synapses receive, respond to, and reenact the rhythms of cinema” (Barker 2009, p. 3). Building upon Marks’ notion of ‘haptic visuality’, Barker’s notion of skin is a touching, caressing gaze concerned with the surface and texture of the cinematic image while musculature is a grasping kind of gaze involved in gesture, comportment and movement. At the level of musculature, a ‘kinaesthetic empathy’ may be evoked through the intertwining of the body of the film and the body of the viewer (Barker 2009, p. 119). Barker contends the type of perception offered by the film’s body via cinematic technique (for example, via dolly, crane and zoom shots, editing rhythms, etc..) bears phenomenological parallels with the human experience of walking, standing, running, flinching, and so on (Barker 2009, p. 77). For Barker, the product of such bodily resonance is a form of empathy that does not necessitate audience identification with a movie’s characters, and this kind of mirroring invites comparison with Vittorio Gallese’s theory of embodied simulation which is explored in detail in Chapter 589.

65 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. The cognitive turn in Film Studies If Film Studies can be said to have undergone affective and corporeal turns it also experienced a cognitive turn traceable to the 1980s, notably with David Bordwell’s Narration in the fiction film (Bordwell 1985) and Bordwell, Janet Staiger & Kristin Thompson’s The classical Hollywood cinema: film style & mode of production to 1960 (Bordwell, Staiger & Thompson 1985). While these are foundational texts90, it was Bordwell’s expounding ‘a case for cognitivism’(Bordwell 1989c, 1990) that set out its broad set of commitments, such as undertaking evidence-based reasoning marked by a clarity of discourse and a turning away from the ‘Grand Theory’ that dominated the 1970s in favour of diverse forms of middle-level, problem-driven research91. Meanwhile, Noël Carroll (Carroll, Nl 1982, 1988b, 1988a) critiqued the field’s dominant paradigm: an amalgam of Saussurean structuralism, Lacanian psychoanalysis, Althusserian Marxism, and Barthesian semiotics which Bordwell pointedly labelled as ‘SLAB’ theory (Bordwell 1989b). Critiqued as doctrinal, Bordwell and Carroll (Bordwell 1996; Carroll, N 1996) launched a rebuttal of the paradigm through their edited volume Post-theory: reconstructing film studies (Bordwell & Carroll 1996).

Conversely, cognitive film theory (CFT) has been critiqued as naïvely reductionist for seeking naturalistic and universal explanations that lack focus on alterity and gendered, enculturated and political difference. This tension belies an epistemological shift that film theory has undergone over the last century, from Münsterberg’s experimental psychology to its current aegis under Cultural Studies. While explanations involving personal and cultural difference were not immediate in early CFT inquiry, it is an indication of the maturing nature of the approach that cognitive cultural studies and cultural neuroscience are recently emerged fields92. From a historical perspective, then, CFT can be understood as a reinvigoration of an approach originated by Münsterberg, and has become an umbrella broad enough to cover not only film and television but computergames and audiovisual media more generally93. It encompasses theorists as varied as those interested in narrative comprehension (such as Ed Branigan (Branigan 1992a), Kristin Thompson (Thompson, K 1999), Murray Smith (Smith, M 1995) and Greg Currie (Currie 1995)) through to those focussed upon biological, ecological and evolutionary approaches (such as Joseph and Barbara Anderson (Anderson, JD 1996; Anderson, JD & Anderson 2005; Anderson, JD & Fisher 1978) and Torben Grodal (Grodal 1997, 2009b)) as well as those experimental psychologists interested in the cognitive and phenomenological dimensions of media aesthetics such as Arthur Shimamura and colleague Stephen Palmer (Shimamura 2013b; Shimamura & Palmer 2011).

Cognitive film theory, it should be noted, is comprised of several undercurrents. The earliest of these exhibit not only a scepticism toward Theory but also a proclivity for visual- and narrative-centric analysis as well as a privileging of ‘hard’ cognition (that is, rationality over affectivity). The cognitive philosophical stance adopted by Carroll (Carroll, Nl 1988b), Bordwell (Bordwell 1989a), Branigan (Branigan 1992a) and Currie (Currie 1995) is emblematic of this ‘hard’ form. In contrast, a later undercurrent admits engagement with cinematic affect albeit with a continued (and unacknowledged) ocularcentrism and narrativism. This undercurrent takes both orthodox and unorthodox manifestations. The orthodox is a revisioning of the aforementioned ‘hard’ form, and considers filmic emotion to be a sub-component of cognition. Theorists working with a predominantly cognitive understanding of affect – that is, the so-called ‘appraisal’ theories of emotion – include Ed Tan (Tan, ESH 1994b, 1994a, 1996), Murray Smith (Smith, M 1995), and Carl Plantinga (Plantinga, C 1993, 2009; Plantinga, CR 1999), and the later work of Noël Carroll (Carroll, Nl 1990, 1997, 1999)). More recently, a radical and alternative understanding of cinematic affect and embodiment has emerged within CFT, exemplified by Torben Grodal’s PECMA flow model (Grodal 1997, 2006, 2009c)) and Greg M. Smith’s mood-cue approach (Smith, GM 1999, 2003b). In simple terms, the ‘hard’ cognitive form of CFT focusses upon top-down processes that generate narrative comprehension and the necessary ‘theory of mind’ required to understand the motivations and behaviour of a story’s characters while the more recently developed ‘soft’ form focusses upon the formal features that activate the bottom-up processes that induct affect into the body of the audience. Plantinga and Smith’s edited volume Passionate views: film, cognition, and emotion (Plantinga, CR & Smith 1999) is an early admixture of ‘hard’ and ‘soft’ attitudes toward cognition and affect in cognitive film theory, mostly from a visual narrative perspective94.

The kind of CFT exemplified in the research of this later generation of which Plantinga, Tan, Grodal, and Smith are a part represents a move away from the tenets of the first wave in cognitive science and toward understandings of the mind as embodied and the importance of the role of affect in that embodiment thus bringing CFT into step with insights from contemporary cognitive psychology and neuroscience. The study of human affect and emotion, and the bodily and mental systems which give rise to them, is complex and littered with false starts and competing theories, and it is for this reason Chapter 4 of this thesis is dedicated to explicating current understandings on the relationship between affect, meaning-

67 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. making and consciousness in the field of emotion science that are relevant to the development of an embodied simulation account of cinematic sound design.

Implications of CFT for cinematic sound. Although early CFT tended to dwell upon vision and narrative it nevertheless opened the way to an empirical aesthetics of sound. An early consequence of CFT for thinking about cinematic sound is a scepticism toward explanations offered by Grand Theory. Trenchant examples include Noël Carroll’s ‘Cracks in the acoustic mirror’ (Carroll, Nl 1996) which censures the psychoanalytic film-sound theory of Silverman’s The acoustic mirror: the female voice in psychoanalysis and cinema (Silverman 1988), and Jeff Smith’s critique of the post- structuralist and psychoanalytic approaches used in theorising the music score’s ‘unheard melodies’ (Smith, Jeff 1996). Similarly, the self-effacing techniques of sound editing and mixing which are explained as manifestations of bourgeois ideology, as proposed by Mary Anne Doane (Doane 1980b), may be approached by a naturalistic account. Rather than understanding an audience as ideologically ‘sutured’ into narrative, an empirically grounded explanation may be mounted that proposes the absorptive fluency of sound design to be analogue to the autonomic processes of self-erasure performed by the human sensory system in the act of perception. Rather than being ‘sutured’, the audience becomes immersed in the perceptual reality of the cinematic world. (Such a proposal is developed in Chapter 4 using Drew Leder’s account of perception and the innate bodily processes that must be necessarily rendered transparent to consciousness (Leder 1990).)

The rise of New Hollywood is also the rise of a diversification and sophistication of the soundtrack leading to the galvanisation of the practice of sound design, particularly with regard to sound effects. Gianluca Sergi, in his ‘A cry in the dark: the role of post-classical film sound’ (Sergi 1998), is possibly the first to focus upon the sonic nature of this form of cinema. Examining sound technology (specifically, the technology of Dolby Stereo) and the techniques that evolved around its use, Sergi shows how New Hollywood reconfigured the relationship between sound and image, and developed new modes of production and reception (Sergi 2004). Moreover, as Mark Kerins (Kerins 2010) observes, the post-classical evolution of sound design which Sergi identifies with the emergence of Dolby Stereo underwent another revolution with the transition from analogue to digital surround sound. At the creative core of post-classical sound is the advancement of technique that heightens and complicates the function of aural sensation and affect, for “audiences are encouraged not just to listen to

68 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. sounds but to ‘feel’ them” (Sergi 1998, p. 162). This leads Sergi to not only advocate the importance of the expressive and visceral capacities of contemporary sound design for scholarly study but also defend the aesthetic value of sound effects (Sergi 1998, 2003, 2006). Where sound effects were once thought vulgar and valueless, Danijela Kulezic-Wilson observes sound design is becoming the ‘new score’ (Kulezic-Wilson 2008). Although Kulezic-Wilson’s privileging of the so-called ‘musical’ qualities (rhythmic, kinetic, affective) of sound design is questionable for its reinstatement of a conventional hierarchy, her observation rings true that the power of contemporary sound design flows from “the cinematic presence embodied in the ‘music of here’ – the sound of a place and moment which envelops and opens itself to the characters in their stillness or continuous movement” (Kulezic-Wilson 2008, p. 128). Strikingly, new sound design practices tend “to increase the aesthetic capacity of sound effects by emancipating them from strictly narrative functions” (Kulezic-Wilson 2008, p. 130).

Increasingly urgent questions concerning the interrelationship of aural sensation, spatialisation, affect and the possible non-narrative functions of sound design form a cluster of problems which force a break with earlier modes of theorising the soundtrack. To better articulate a programme of research for cinematic sound that includes all its aspects Mark Kerins suggests an agenda focussed upon the areas of surround sound; production practices95; the relationship between ‘sound’ and ‘music’96; and the study of the soundtrack as a unified object (Kerins 2008).97

Kerins (Kerins 2006, p. 41) notes the evolving nature of techniques and technologies in the production of surround soundscapes has gone largely untheorised although recent scholarship is beginning to redress this imbalance. Aside from Sergi’s pioneering work is the research of Kerins’ himself. Kerins has produced several significant explorations of digital surround sound and its impact upon cinematic narration and aesthetics (Kerins 2006, 2010, 2013b, 2013a). A key concept to emerge from Kerins’ work is the sonic ultrafield, discussed above, which critiques and updates Chion’s notion of the superfield. Christos Manolas and Sandra Pauletto (Manolas & Pauletto 2009), Johan-Magnus Elvemo (Elvemo 2013), and Damian Candusso (Candusso 2016b, 2016a) have also expounded upon the role of surround sound and spatialisation in the act of narration, with Elvemo taking a decidedly cognitive approach. Beth Caroll, however, takes a different tack. Dissatisfied with narrative as the dominant framework for understanding cinema, Caroll argues for a spatial methodology that centres on the

69 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. aesthetic, embodied and nonconscious dimensions of Gestalt perceptual organisation. In Feeling film: a spatial approach (Carroll, B 2016), Carroll explores aural cinematic space through the phenomenology of Merleau-Ponty and the haptics of Sobchack, Laura U. Marks and Jennifer M. Barker (Carroll, B 2016, pp. 49–87).

Although an exploration of the relationship between sound, sensation and affect is under- defined in Kerins’ agenda, post-classical sound theory is beginning to ask questions of the relationship between material surface and style and the sensation and affect produced. William Whittington (Whittington 2007, 2009, 2013), for example, focuses upon the corporeal and affective dimensions of the immersive spectacle of digital surround sound, and Lucy Fife Donaldson (Donaldson 2017a, 2017b) focuses upon the affective potential of sound effects, music score and the sound mix. Following Sergi and Kerins in turning to the praxis and implicit knowledge of sound design professionals, Donaldson rightly observes “sound practitioners frequently speak about their work in terms of feel, texture and experience” and she emphasises sound design as a mode of communication brought about by “affective and empathetic connections between the experience of the sound designer and audience” (Donaldson 2017a, pp. 31 and 44). Importantly, Donaldson understands the “affective qualities of surface as expressively produced by these various components of soundtrack in order to highlight the important connection between sound and feeling … Attention to surface therefore enables consideration of the film world as designed and as affective” (Donaldson 2017b, p. 85, emphasis in original). Although the above signals a movement in scholarship toward exploring the affective nature and function of cinematic sound the fundamental mechanisms responsible for creating sonically induced affect remain undeclared.

Cognitive film theory possesses a growing literature examining the relationship between cinema and affect. As noted above, the role of emotion is being investigated in the context of narrative cinema by Ed Tan (Tan, ESH 1994b, 1994a, 1996) and Carl Plantinga (Plantinga, C 2009; Plantinga, CR 1999) among others, and Robert Sinnerbrink is exploring the ethical and moral functions of mood (Sinnerbrink 2012, 2016). Reflecting research of cinematic affect at the level of the bodily and autonomic is the PECMA flow model of Torben Grodal (Grodal 1997, 2006, 2009c) which maps nonconscious processes in the production of narrative consciousness, the emotion communication model developed by Anne Bartsch and Susanne Hübner (Bartsch 2004; Bartsch & Hübner 2005) which incorporates neuroscientific and social constructionist theories, and the mapping of film structure in relation to the human emotion

70 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. system by Greg M. Smith (Smith, GM 1999, 2003b). What is under-explored, even among these works, is a fine-grained explanatory mechanism of cinematic sound and affect. In contrast, the cognitive psychology of Annabel Cohen and Scott Lipscomb in music score and of Kathrin Fahlenbrach in sound design are significant exceptions.

Cohen’s Congruence-Association Model of film music (Cohen 2013b, 2015), developed over a three decade period, has yielded insights into the role of music in the comprehension of filmic narrative (Cohen 1998, 2005, 2013a, 2014a). Such a cognitive approach to film-music is pertinent to this thesis because score typically exists outside the diegesis and yet it heavily influences audience perception of protagonists and narrative comprehension. Lipscomb provides a complementary model of multimedia cognition that examines the role of audiovisual accent structures, their alignment, and their role in focussing audience attention within the audiovisual scene (Lipscomb 2013). On Lipscomb’s account, ‘accents’ are moments of salience occurring in either aural or visual modalities, and so this research revises and supercharges Chion’s notion of ‘synchresis’ by exploring in detail the nature and perceptual impact of audiovisual synchronisation and multimodal integration (Lipscomb 1999). Another important insight of Cohen’s Congruence-Association Model for this thesis is her explication of music as a source of ﬁlmic emotion (Cohen 2001).

Kathrin Fahlenbrach (Fahlenbrach 2002, 2005b, 2005a, 2008, 2016a, 2016b, 2017) argues audiovisual media cannot be fully comprehended using established narrative and linguistic paradigms, and has pioneered an affect-centric model of filmic sound design. Like Cohen and Lipscomb, Fahlenbrach begins with the mechanisms of crossmodal interaction that sit at the heart of designing cinematic imagery, but focusses upon the synaesthetic mappings that produce audiovisual metaphor. Following George Lakoff and Mark Johnson who propose conscious thought to be rooted in unconscious mechanisms that cross-map embodied experience, Fahlenbrach applies conceptual metaphor theory to the context of cinema. Moreover for Fahlenbrach, an associative network of affect and emotion is simultaneously produced through such synaesthetic processes to create forms of embodied meaning. Fahlenbrach’s work impresses that to understand the design of cinematic imagery one must grasp the role of the human emotion system in forming perception and guiding narrative comprehension, and it is for this reason her work is explored and built upon in Chapter 3.

71 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Post-classical spectacle and narrative The turn to affect and the corporeal returns us to a faultline that has long run through theories of film. Where the cinema of attractions stressed sensation and spectacle over narrative, the qualitative shift of the New Hollywood appeared to do the same. The heightened qualities of this form of cinema attracted labels like ‘high-concept’ and ‘blockbuster’ (Wyatt 1994), and according to Richard Maltby the shift is characterised by movies that have

little commercial investment in narrative except as a vehicle for the movie’s other pleasures, and equally little commitment to a classical hierarchization of a narrative system over those of space and time. (Maltby 1998, p. 39)

Maltby’s observation puts two features into stark relief: first, the de-prioritisation of narrative coherence in formulating cinematic experience; and, second, the stylistic elevation of spectacle. The aggressive form of this contemporary aesthetic – Geoff King’s ‘impact- aesthetic’ (2000, pp. 102 and 168) – reconfigures cinematic space and time to distinguish post-classical from classical cinema. This distinction echoes Gunning and Gaudreault’s identification of ‘attractions’ in early cinema to distinguish it from the later dominance of narrative in classical cinema. Thomas Elsaesser (1998), Geoff King (2002) and Eleftheria Thanouli (Thanouli 2006, 2008, 2009) have each participated in advancing descriptions of a post-classical style that stresses sensation over narrative coherence. However, the proposal of a ‘post-classical cinema’ remains contentious. Murray Smith (1998), Katherine Thompson (1999) and David Bordwell (Bordwell 2002, 2006) dispute the accuracy and utility of the term for they consider contemporary cinema to be continuous with the classical in narrative terms, and argue differences in style do not annul this continuity. Jeff Smith sums up the stance by observing:

contemporary filmmakers may employ shiny new cinematic devices and techniques, but they are firmly committed to the principles of narrative unity, clarity, and coherence that have governed the tradition of classical cinema for nearly a century. (Smith, J 2013, p. 332)

In short, the value produced by narrative is considered paramount to that produced by the sensation of cinematic attractions. This stance is in keeping with the historical framing of spectacle as superficial and, more damningly, “a suspect element whose ideological

72 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. operations infect the film viewer unaware” (Grindon 1994, p. 35). Leger Grindon traces the conception of spectacle as ‘untrustworthy image’ from Debord98 into Lacanian psychoanalysis and thus into 1970s film theory via the writings of Jean-Louis Baudry and Laura Mulvey (for example, ‘Ideological effects of the basic cinematographic apparatus’ (Baudry 1986) and ‘Visual pleasure and narrative cinema’ (Mulvey 1975)). Grindon claims the political critique of spectacle has hindered precise thinking about the cinematic image by pointing to anti-illusionist attempts to cauterise the audience’s relationship to the image’s material qualities from the narrative content of the image (Grindon 1994, p. 37). It is against this backdrop that Bordwell’s proposition of post-classical style as intensified continuity can be seen as a form of rehabilitation that seeks to subsume spectacle into the machinery of narrative.

Bordwell refutes the claim of a narratively incoherent post-classical cinema by making the counterclaim that contemporary techniques are essentially classical filmmaking principles “amped up”. He terms this phenomenon intensified continuity (Bordwell 2002). What is ‘intensified’ by post-classical style are the techniques for narrativisation already active in classical cinema. Intensified continuity, on Bordwell’s account, is comprised of at least four visual characteristics – faster picture editing rates; extreme uses of short and long lens lengths; close framings in dialogue scenes; and more free-ranging camera movements99. Roughly, these characteristics boil down to heightened control over tempo and subjective/objective space (via limiting of visual planes, and mobile camera). Rather than produce narrative incoherence, Bordwell asserts, “intensified continuity has endowed films with quite overt narration” (Bordwell 2002, p. 25). From Bordwell’s perspective, these characteristics constitute no interruption to classical cinema because the audience is asked to “revel in still more flamboyant displays of technique – all the while surrendering to the story’s expressive undertow” (Bordwell 2002, p. 25).

Bordwell’s conception of intensified continuity is ocularcentric, but Jeff Smith (2013) and Amanda McQueen (2013) have applied it to the soundtrack. Smith (2013, p. 338) catalogues six dimensions of post-classical style that constitute what he considers an aural counterpart to Bordwell’s concept; namely, (i) increased volume; (ii) low frequency effects; (iii) expanded frequency range; (iv) sound spatialisation; (v) hyperdetailed Foley effects; and (vi) nondiegetic sound effects used as stylistic punctuation.

73 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Although these dimensions are conspicuous qualities of contemporary cinema, Smith does not provide an altogether effective account as to why these techniques are continuous with classical narrative cinema. Increased volume, in Smith’s description, is shorthand for a number of aspects related to the capacity of contemporary audio technology to deliver greater dynamic range with lower distortion. Technologies, such as those developed by Dolby Laboratories and Sony, expand the dynamic range of the soundtrack to produce louder and less distorted sound peaks but also exceptionally detailed yet quiet passages which would otherwise be lost into the noise floor of pre-Dolby soundtracks100. Smith notes increased volume has heightened visceral and emotional effects, but he does not explore the value of these effects nor explain his assumption that these effects advance narrative. Thus, given the claim “Dolby is simply an enhancement of already existing principles of Hollywood sound design” (Smith, J 2013, p. 339), it is unclear how volume, in and of itself, is a classical narrative principle.

Smith reiterates a common analogy between contemporary movies and ‘rollercoaster rides’, and correctly identifies the role of low frequency effects (LFE) in the production of this type of spectacle. Historically, he traces the use of LFE to spectacle films of the 1970s. In light of Gunning and Gaudreault’s research concerning the non-narrativity of cinematic attractions, Smith’s argument seems to annul itself, and it remains unclear in which ways LFE aids narration. The movies cited by Smith are war and action-adventure genre movies, and this suggests an objective of LFE techniques is the induction of a specific kind of affect in the audience as much as a representation of threat to characters in the diegesis. Even so, this begs an explanation of the relationship between inducted versus represented affect, and how sonic affect functions in relation to narrative.

Smith identifies expanded frequency range as a characteristic of post-classical style, but, like his reference to LFE, it remains unclear why this should be considered continuous with classical narrative cinema. Smith cites the adoption in the 1930s of the Academy curve as a method to standardise soundtrack reproduction, but, as Smith notes, the curve enforced an extreme roll-off in the low and high frequencies. In effect, the Academy curve took the limitations of the technology as a base line to determine what should (and should not) be designed into the soundtrack. What the limitations of the Academy curve privileged was the human voice101. Thus, it is unsurprising classical cinema privileges dialogue as a narrative

74 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. tool. All this aside, it remains unclear why an expanded frequency range is a continuation of classical narrative technique.

Taken together, the technical capacity for greater dynamic range, lower noise floor, and wider frequency range – including the subsonic qualities of LFE102 – provide sound editors and mixers with tools to create substantially more defined and texturally nuanced sound objects and events. In other words, what arises is ‘hyperdetail’. Although Smith focuses upon Foley when exploring the phenomenon, the capacity for hyperdetail is available across the entire fabric of the soundtrack. When high-resolution sound objects and events are coupled with a heightened control over proximity and spatialisation what is produced are perceptually realistic soundfields. These soundfields are the aural identity of the cinematic environment. In their analysis, Smith and McQueen do not speak specifically of hyperdetail in terms of perceptual realism but, in following Bordwell, consider it an aspect of ‘world-building’ under the rubric of narration. But is narration the sum total of what is produced in these cinematic processes?

Reflecting upon the tension in early cinema between spectacle and the emerging capacity for narration, Gunning observes “[a]ttractions are not abolished by the classical paradigm, they simply find their place within it” (1993, p. 4). The same can be said of the attractions nested within post-classical cinema. Simon Lewis (2014) – following Geoff King (2000) and Yvonne Tasker (2004) – proposes spectacle is not best understood as completely devoid of narrative content. For Lewis, spectacle resists analysis because it has been approached solely from the normalised perspective of narrativity (Lewis, S 2014, p. 217), and instead sets out a ‘transmission’ model to encapsulate both spectacular and narrative drives. What is transmitted in this model is ‘information’. On Lewis’ account, the experience of a movie is the experience of a flow of information, but what constitutes information is somewhat under-defined. Lewis posits information may be visual or aural, and “may be directly relevant to the plot or it may be incidental to it”. Lewis identifies two modes of transmission: narrational transmission and non-narrational transmission, which are defined thus:

Narrational transmission is that element of transmission that is aimed solely at providing the spectator with information about the plot, directing the spectator’s attention to the elements of the film that advance the narrative. Nonnarrational transmission is everything else… (Lewis, S 2014, pp. 216–217)

Again, the proffered differentiation is under-defined, but Lewis’ intention is to conceive of the two modes of transmission as commensurable (that is, both transmit information of some kind) and thus place them within a single system. As terms, narrative and spectacle cease to identify discrete sequences within a movie but are recognised as a shifting equilibrium between modes of information transmission occurring within the same sequence (Lewis, S 2014, p. 217). Echoing Gunning’s observation, Lewis proposes spectacle is “not something that lies outside or interferes with the narrative drive and a narrative sequence will contain many elements that contribute toward the spectators enjoyment of the film but that are unrelated to the narrative” (Lewis, S 2014, p. 217). In examining the lobby shoot-out sequence from The Matrix (Wachowski & Wachowski 1999), Lewis reveals spectacle and narrative to function as components of the same process, and thus “contradict the notion that narrative is somehow superior to spectacle”.

Lewis adds depth to the model by parsing the concept of spectacle into two types which he terms event spectacle and object spectacle. In event spectacle

things happen to the characters, often putting them at risk of death or serious injury, and it is thus intended to work with the narrative and to increase emotional impact of the film … Event spectacle is cued strongly within the film’s narrative architecture … Event spectacle tends to generate excitement, astonishment, awe, possibly fear, arising out of the spectator’s identification not only with the characters’ predicament but also the event in itself. (Lewis, S 2014, pp. 217–218, emphasis added).

Object spectacle, in contrast, “depends less on the context of its narrative structure and more directly on the presentation of the object as spectacle in itself” (Lewis, S 2014, p. 220). Object spectacle is a

stylistic adjustment to achieve an emotional effect, a more subtle shaping or manipulation of the transmission than event spectacle. It may, like event spectacle, employ cueing to create an alignment of the spectator’s emotions with the characters on the screen, but often it is not overtly part of the narrative structure of the film … If event spectacle represents an exaggeration of elements of the mise-en-scène, object spectacle is more of an emphasis of those elements. (Lewis, S 2014, p. 220).

Event and object spectacle, therefore, can be understood as mechanisms for the production of various kinds of affect which may or may not be produced as a consequence of narration. Indeed, the somewhat foregrounded role of affect in Lewis’ model differentiates it from Bordwell’s cool and cognitive understanding of intensified continuity. However, Lewis’ clustering of astonishment, awe and fear in the explication of event spectacle – those very qualities which mark the sublime – suggests the transmission model requires further development to fully apprehend and integrate the function of affect. Event spectacle is concerned with an audience’s emotional identification with a character, but also with the audience’s response to the event in itself. Hidden inside this description are two types of affective response. The first concerns registration (leading to possible sympathy or empathy) by the audience of the represented emotions attributed to a character. The second response concerns what Suzanne Keen (Keen 2011, 2015) calls ‘narrative emotions’ which are those affects which go beyond character identification and are elicited in response to the scenario. Object spectacle, on the other hand, is the induction affect in the audience as it encounters the materiality of audiovisual imagery. The affect inducted by object spectacle arises from a direct experience of the perceptual qualities of the object, and is therefore of a more ecological nature. What is interesting about object spectacle, although it is elided in Lewis’ discussion, is this form of affect arises non-narratively and yet appears capable of being targeted onto the characters in the diegesis. This phenomenon and its relation to James A. Russell’s concept of core affect is discussed in Chapter 3.

Lewis’ transmission model provides a plausible illustration of how cinematic spectacle functions, and why conceiving of spectacle and narrative in hard oppositional terms is not useful. The model proposes entwined narrative and non-narrative processes for transmitting information, and hints at the common affective basis of this information. Within the transmission model differentiation is made between spectacle which progresses narrative and spectacle which is independent of narrative drive. Under-examined in the model are the functional differences between affect that is represented and that which is inducted. Lastly, the transmission model of spectacle reveals the limitations of Smith and McQueen’s application of Bordwellian intensified continuity. Rather than the subsumption of post- classical sound design and spectacle as aspects of classical narrativity, spectacle and narrativity are shown to co-exist within a larger system of meaning production. Paradoxically, Smith recognises post-classical sound and visual strategies work in tandem “insofar as they

77 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. function similarly to heighten the affective, sensory, and phenomenological dimensions of contemporary film style” (Smith, J 2013, p. 335) yet there is no attempt to explore these dimensions outside their possible contribution to narrativity. What is under-explored in Smith’s aural counterpart to intensified continuity is the cinematic impulse toward the production of affect, immersion, and presence.

Aural spectacle Aural attractions exist along a spectrum. Often thought of as the bravura of a dazzling sonic event, aural attractions may also manifest as small but salient moments which intimate the beauty of the world. The high-octane action of Mad Max: Fury Road (Miller, G 2015) is an example of the former while the delicate environmental soundscapes of The Revenant (Iñárritu 2015) are examples of the latter. These conceptions of aural spectacle are at polar extremes but what is common are prominent qualities of immersion and presence.

Immersion and presence are at the heart of the sound aesthetic developed by Alfonso Cuarón and his sound designers Richard Beggs and Skip Lievsay in Children of Men (Cuarón 2006), Gravity (Cuarón 2013b) and Roma (Cuarón 2018). These movies incorporate both subtle aural attractions and narrative-freezing moments of spectacle, and as a progression of creative works they signal a concerted move away from the classical conception of cinematic aural space. In Roma, Cuarón’s use of the Dolby Atmos103 platform creates not the impact aesthetic of a typical blockbuster (although there are passages of intense but subtle aural spectacle) but an immersive aural space which envelops a more-or-less static camera. Contrary to orthodoxy, Cuarón stresses

the real function of this tool [Dolby Atmos] is going to be in intimate films, not in this big loud tentpoles … It’s about spread, and the beautiful thing of spreading all these sounds, and being able to be very geographically specific … (O'Falt 2019)

Cuarón’s aesthetic use of Dolby Atmos signals an ebbing of the classical legacy in sound mixing which orients audience attention to the front of the auditorium and glues dialogue (and music score) to the screen. Cinematic surround sound in this mode functions in the way audition functions to cue visual attention in the real world, and is aligned with the techniques for designing sound space in computergames where an unencumbered use of surround sound has been in use since the early 2000s (Kerins 2013b).

The spectacle of perceptually realistic soundfields also serves in the production of affect and presence. Examining the functions of multichannel sound, Daniel Västfjäll (2003) has shown perceptual immersion to be highly entwined in the production of affect and presence. When equipping an auditory virtual environment (AVE) with mono, stereo and 6-channel surround soundfields,

stereo and six-channel reproduction resulted in significantly stronger changes in emotional reactions than the mono condition. Further, six-channel reproduction received the highest ratings of presence and emotional realism. Taken together, the result suggested that both emotional reactions and ratings of presence increase with spatialized sound. (Västfjäll 2003, p. 181)

In examining the concept of a distinctly aural attraction, Teresa Forde (2011) observes such attractions have the unique capacity to resist absorption into the visual image by concretising phenomenological experience and thus articulate “what is visually non-representable” (Forde 2011, pp. 78 and 82). Analysing the sound design and music score of Sunshine (Boyle 2007), Forde identifies the movie’s aural attractions are tied to the sublime, and manifest as strong emotional experiences such as wonder, awe, fear and hope (Forde 2011, p. 72). Although Forde stops short of interrogating aural attractions in terms of ‘aesthetic chills’, her analysis abuts insights from the contemporary psychological research of aesthetics which considers this affective cluster (awe, fear, joy) to be central to a pan-cultural response (Konečni 2005). From a music cognition approach, chills are intense but transient emotional and psychophysiological responses to an aesthetic experience, and appear to be an embodied response to significance104. Not only does sonically induced chills possess a recognisable psychoacoustic profile (Grewe et al. 2011; Grewe, Kopiez & Altenmüller 2009; Grewe et al. 2007; Grewe et al. 2005; Nagel, F et al. 2008) but it is posited they arise from evolutionary adaptations which favour auditory learning, facilitate an affective signalling system which predates the emergence of language, and, over time, came to perform social functions (Altenmüller, Kopiez & Grewe 2013b, 2013a). Aesthetic chills have also been linked to curiosity and cognitive dissonance experienced during knowledge acquisition (Schoeller 2015). What is particularly significant to theories of cinematic spectacle and aural attraction is the proposal that aesthetic chills are “a universal marker of openness to experience” (McCrae 2007).

Presence is an under-researched phenomenon within Film Studies (Neuendorf & Lieberman 2010). Although central to Gunning and Gaudrault’s concept of cinema-of-attractions (Gunning 1986) and allied to Gaudrault’s theory of monstration (Gaudreault 1987, 2009) it remains ill-defined. Oftentimes, immersion is erroneously conflated with the psychoanalytic film theoretical concept of ‘suture’. This lacuna is conspicuous when one considers the history of cinema is also a history of “striving for an ever-greater level of presence through technological innovation, changes in aesthetic form, and developments in narrative structures and performance styles” (Neuendorf & Lieberman 2010, p. 9). Kimberly Neuendorf’s claim inverts an assumption central to intensified continuity; narrative is not an inexorable final product but is itself subsumed in the production of presence. This aligns with Lewis’ transmission model which proposes narrative and spectacular imagery are involved not in a zero-sum tension but are components of a larger system.

2.4 Framing the gap in Film Studies – beckoning fields of innovation The foregoing review of literature reveals gaps and contestable assumptions within Film Studies concerning the nature and function of mediated sound. Although Film Studies is undergoing a somatic turn, a legacy persists which frames the function of sound to be the support of vision in the support of narrative. On this standard account, as the literature has shown, sound has long been circumscribed by a hierarchical logic of visual narrativism which privileges a linguistic and cultural determination of meaning, and thus there is not a well- established, historically-embedded conceptual framework by which to recognise and explain the full array of phenomena and functions of cinematic sound. The concept of intensified continuity is illustrative of this kind of visual narrativism which subsumes aural spectacle as a form of narrative. The history of cinematic media is a history of the impulse toward perceptual and psychological immersion, toward both the feeling of ‘being there’ and of understanding the mind of a mediated character. Since the 19th century, the sound design techniques and technologies which produce immersion and presence have undergone sustained and radical development yet the function these mechanisms fulfil remain under- theorised within Film Studies. The standard paradigm of visual narrativism does not account for everything that is occurring within the designed sound of a cinematic experience, and the disjunction becomes ever more apparent as cinematic media move deeper into a post-

Theorising cinematic sound, then, requires an innovative and interdisciplinary approach (Kerins 2008; Langkjær & Have 2012). In the context of this thesis, the relevant themes of research are interspersed through multiple disciplines but often these disciplines share little in the way of a common terminology or conceptual framework. As Langkjær & Have note:

Sound studies covers a vast area of different fields within art studies, architecture, musicology, film and media studies, perceptual psychology, anthropology etc. This not only means that sound has no natural ‘home’, but also that sound is conceptualised in various ways depending on the concepts, methods and approaches that traditionally define each area of study … It is not possible to define a common language, a lingua franca, spoken by all researchers and practicians of sound … such a language may not even be desirable, since we can benefit from the present polyphony, which keeps sound studies connected to and integrated with all kinds of academic and artistic practices. (Langkjær & Have 2012, p. 2, italics in original)

Interdisciplinarity, then, is charged with advantages and disadvantages. The advantages are apparent in the wealth of conceptually and methodologically specific knowledge that has amassed across a multitude of academic fields. The disadvantages arise when attempting to put this fragmented knowledge in dialogue. Langkjær & Have appear comfortable with the ‘polyphony’, but the connectedness and integration may not be as firm as they suggest, for this babel also presents a friction to theory-making.

In Chapter 1.7, I identified the methodology of this thesis to be interdisciplinary in spirit because it adopts an embodied cognition approach to explain the nature and function of cinematic sound design. Mapping insights I have gained through my own creative practice against cognate concepts emerging from the cognitive sciences and re-mapping these concepts into a broader humanities context, the method recognises art and science to be complementary modes of knowing.

Interdisciplinarity of this kind suggests a consilience between the so-called ‘two cultures’ even as it remains a fraught issue. Michael Bull, Paul Gilroy, David Howes and Doug Kahn, for example, assert the “perceptual is cultural and political, and not simply (as psychologists and neurobiologists would

81 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. have it) a matter of cognitive processes or neurological mechanisms located in the individual subject” (Bull et al. 2006, p. 5), and as a statement on sensory research in the humanities and social sciences their stance bristles with a desire to reinforce disciplinary boundaries by casting the endeavours of those scientists at the humanities/sciences nexus as naïvely simplistic. Such tension is similarly manifest when Martin Jay warns that consilience is “the hostile takeover of the humanities by the natural sciences” (Jay 2002, p. 276) and urges the disciplines to remain unreconciled (Jay 2011) even as he argues visuality is unlikely to be culturally coded ‘all the way down’.

I accept consilience to be controversial in some quarters, and am guided by Ione when she maintains artistic practice is well situated to confront the schism (Ione 2006). I am similarly persuaded by Edward Slingerland and Mark Collard (Slingerland 2008a, 2014; Slingerland & Collard 2011a) in their efforts to frame a ‘second wave consilience’. In such a context, indicating the emergence of a ‘third culture’ approach to film theory, Murray Smith (Smith, M 2011, 2017) has developed a strategy he terms triangulation. The strategy of triangulation resonates well to the issues of sound design that I have been setting up in the first two chapters of this thesis. On Smith’s account, triangulation seeks explanation of aesthetic experience by the intersection of three types of evidence:

the phenomenological level (what, if anything, it feels like when we undertake some mental act), the psychological level (what sorts of psychological capacities and functions our minds seem to possess), and the neurological level (what seems to be happening in the brain when we exercise these capacities or have these experiences). In other words, we have evidence pertaining to our experience of mental phenomena; the information processed by mental activity; and the physical realization of the mental … Where any two, or all three, forms of evidence mesh in this way, so each of them is corroborated. (Smith, M 2011, p. 83)

Smith’s method makes explicit the spirit of interdisciplinarity I have brought into play when developing my own embodied cognition approach to cinema (Ward, MS 2015). However, where Smith’s deployment of triangulation gravitates to the higher functions of narrative and character psychology my thesis attempts to remain focussed at the subpersonal and nonconscious levels of embodied meaning actuated by cinematic sound design. Such is the mission of the remainder of this thesis, now that I have mapped the ‘thickness’ and ‘thinness’ of a century or more in cinematic sound studies.

82 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. In propounding an interdisciplinary spirit, the literature review that I have presented encompasses several disciplines outside Film Studies. Indeed this inter-disciplinary reach will continue in the upcoming chapters, as I examine (in Chapter 3) the function of crossmodal interaction via synaesthetic and affective mechanisms, all of which can be leveraged in the cinematic design of immersion and embodied meaning; then in Chapter 4 I will synthesise the relevant neuroscientific theories of affect and embodied cognition. These extensions to the main literature review, which have been presented in this current chapter, will be necessary for advancing my proposed interdisciplinary, embodied simulation account of cinematic media. Moreover, because most extant theories of film tend to examine narrative cinema, and because my embodied simulation account resides at distinctly nonconscious and subpersonal levels, I feel bound to signpost how such an embodied simulation account can interface with narrative-centric film theory. Thus, Chapters 3 and 4 will provide theoretical grounding for Chapter 6 which will explore how affect is central in the generation of consciousness and narrative. As Chapter 6 will amalgamate much of the interdisciplinary surveying and synthesising work that I have enacted in the preceding chapters, my contribution to deepening the understanding of the emerging field of “embodied cognition and cinematic sound design” will become clearer and clearly well-evidenced there.

But first, we need to examine the implications of sound within systems of human bodily affect in order understand better how cinematic sound design can engage with such embodied experience. This is the mission of the next chapter.

------this page intentionally blank ------

3.1 The status of sound in conventional Film Studies The review of literature in Chapter 2 identified four layers of conceptual straitjacketing which act to disenfranchise cinematic sound from productive theory-building. To recap, the first straitjacket is an assumption that vision exercises hegemony over audition; the second is an assumption that rationality commands dominance over emotion; the third is a belief that meaning is the dominion of language and culture; and the fourth is an assumption that consciousness and narrative command significance over the body’s sensorimotor system and its associated imagery. A gap emerges in the literature when these tendencies are identified in the classical paradigm of film theory. However, this is no small ‘gap’. It is the absence of a coherent conceptual framework by which to explain the structure, function and experience of cinematic sound. The purpose of the current chapter is to address this lack by beginning afresh a systematic building of a conceptual framework.

In returning to fundamentals we return to the Bazinian question ‘What is cinema?’ Firstly, this chapter sets about developing a working definition of cinematic media. It does this by interrogating the tacit knowledge upon which sound design praxis is based. This definition centres upon a broadly construed concept of imagery. Within Film Studies the term ‘image’ almost always denotes visual image. This may be conventional but it is misleading for it is insensitive to the existence of non-visual forms of imagery such as auditory, motor, emotional, and, of course, mental imagery. I take image to be directly related to the concept of imitation: as Barthes notes, “[a]ccording to an ancient etymology, the word image should be linked to the root imitari” (Barthes 1977c, p. 32, emphasis in original). An image, therefore, is not necessarily nor exclusively a product of vision. Secondly, the chapter describes the qualities of the cinematic and lays out its affective nature. Thirdly, the terms perceptual design, narrative design, proto-narrative, and Feeling of Body are introduced into the working definition of cinematic media. Perceptual design and proto-narrative build upon notions of the multimodal and affective nature of the cinematic. These terms describe how nonconscious affective processes are activated by cinematic media to produce a Feeling of Body and, as a consequence, provide the basis for narrative proper.

3.2 Praxis and tacit knowledge The careful reader will notice my tendency to speak of ‘cinematic media’ rather than ‘cinema’. This allows a cautious approach to the question ‘What is cinema?’ by asking ‘What is cinematic about cinema?’106 Customarily, this question is answered in visual and narrative terms. Visual narrativism is the dominant model within Film Studies yet it is a paradigm I find profoundly restrictive as a sound designer because it explains little about what I do that may be squared with professional praxis. Sound designers rarely theorise openly about their work, yet it is impossible to perform creatively without some sort of hypothesis as to how movies function107. This knowledge is constantly under test and revision with each movie acting as a laboratory for tacit theory-making. Taken together, these lessons lead to a quite different set of assumptions to those of visual narrativism. In the interests of transparency, I have distilled three key assumptions from my professional practice concerning the nature of cinematic media108. Firstly, sound modifies visual perception, and vice versa. Sound has the capacity to shape visual perception and steer visual attention, and may interact with vision to produce synaesthetic experience. This crossmodal interaction is complex and dynamic, and leads to the assumption that, secondly, cinema is not per se a visual medium but multimodal. What is cinematic about cinema is not moving pictures but moving imagery. Thirdly, because one may have a satisfying cinematic experience without a concomitant experience of narrative, I assume cinema is not axiomatically narrative but affective in nature. Indeed, not all cinema is narrative cinema, and it is unlikely one may have a meaningful narrative experience without it also being an emotional one109.

3.3 A working definition of cinematic media What defines media as cinematic? Human beings perceive the world within a spatio-temporal framework, and the perception of “naturalistic events relies on the ability to integrate information from multiple sensory systems” (Roudaia et al. 2013, p. 1). The cognitive psychologist Stephen Handel observes that “space and time are intertwined frameworks for auditory and visual events” where “spatial and temporal change defines the properties of events and objects” and, “[m]oreover, the spatiotemporal variation of auditory and visual events gives them meaning” (Handel 1988, p. 315). If the basic stuff of cinema is re-imagined as a dynamic set of spatiotemporal relationships, its putative visual primacy gives way to a

Handel (Handel 1988) notes both audition and vision are capable of processing spatial and temporal information, and argues against the hard dichotomy expressed in the dictum ‘space is to time as vision is to audition’. However, legitimate generalisations may be made about these modalities111. For example, when engaged in ambiguous perceptual tasks, such as audiovisual illusions, vision outperforms audition in spatial processing while audition outperforms vision in temporal processing (Repp & Penel 2002). In coarse terms, vision’s strength tends to lie in structuring space while audition’s strength tends to lie in structuring time. These propositions are central to the modality appropriateness hypothesis which postulates:

… the modality that is most appropriate or reliable with respect to a given task is the modality that dominates the perception in the context of that task. Vision has a higher spatial resolution, hence its dominance in spatial tasks, whereas audition has a higher temporal resolution, hence its dominance in temporal tasks. (Shams, Kamitani & Shimojo 2004, p. 28)

As a basic working definition, what makes cinema cinematic is an ability to dynamically mediate an experience of space, time and movement. In recognising cinema as a spatiotemporal system its intrinsic multimodality is foregrounded but, in important addition to this, cinema is an affective spatiotemporal system.

Tacit knowledge drawn from praxis, of course, demands close inspection and, at first blush, these professional intuitions appear contentious and counterintuitive if not extravagant. Nevertheless, evidence is accumulating within the cognitive sciences to support the claim of cinema’s intrinsic affective multimodality, and it is this claim I elaborate upon in this chapter.

3.4 Multimodal integration and cinematic imagery Film Studies conventionally holds silent moving visual imagery to be necessary and sufficient to a definition of cinema whereas sound is merely an “incidental accretion” (Currie 1995, p. 3). The maxim ‘cinema is a visual medium’ authenticates the belief that it is enough to

87 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. experience a cinematic work through vision alone. Although such a claim is commonplace it oversimplifies the cinematic experience. A broad range of empirical evidence indicates human sensory modalities interact with such a high degree of complexity it is untenable to suggest they function in isolation (Ghazanfar & Schroeder 2006; Shimojo, Shinsuke & Shams 2001), and this evidence has far-reaching implications for media theory.

The fallacy of silent vision The concept of autonomous vision is faulty and misleading. Firstly, silent moving visual imagery crossmodally elicits auditory imagery. Auditory imagery112 is the “subjective experience of hearing in the absence of auditory stimulation” (Kraemer et al. 2005, p. 158), and may be reliably activated via mental imagery and rules of expectancy (King, AJ 2006). Raij, McEvoy, Makela & Hari (Raij et al. 1997) found the observation of a visual scene with expected sounds omitted is capable of activating the neural pathways involved in auditory processing. Similarly, Voisin and colleagues (Voisin et al. 2006) have shown that even merely searching for sound in silence activates the auditory cortex, and Hoshiyama, Gunji & Kakigi (Hoshiyama, Atsuko & Kakigi 2001) found the observation of silent moving pictures prompts the retrieval of auditory imagery. Silent moving visual imagery appears to violate fundamental rules of event perception to such a degree it triggers mandatory auditory search (Kraemer et al. 2005; Mustovic et al. 2003). Findings such as these indicate a neuropsychological mechanism is at work whereby vision initiates auditory imagery to ‘fill- in’ missing aspects of an informationally degraded perceptual event113.

Perceptual filling-in appears to be neither uncommon nor uni-directional. For example, Väljamäe & Soto-Faraco (Väljamäe & Soto-Faraco 2008) found that sound may fill-in missing aspects of visual motion. Thus, if sensory information is degraded in a particular modality, it may be synthesised, within limits, from another modality to round out event perception114. Our perceptual system seeks wholeness through crossmodal interaction, and transgression of its innate rules entails adverse reactions. From such a perspective, Maxim Gorky’s ambivalent (if unfavourable) response to early cinema is therefore unsurprising. In July of 1896, Gorky wrote of the imperfections of silent moving pictures:

Last night I was in the Kingdom of Shadows. If you only knew how strange it is to be there. It is a world without sound, without colour. Everything there – the earth, the trees, the people, the water and the air – is dipped in monotonous grey. Grey rays of

88 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. the sun across the grey sky, grey eyes in grey faces, and the leaves of the trees are ashen grey. It is not life but its shadow, it is not motion but its soundless spectre. Here I shall try to explain myself, lest I be suspected of madness or indulgence in symbolism. I was at Aumont’s and saw Lumière’s cinématograph – moving photography. (Gorky quoted in Popple & Kember 2004, pp. 2-3)

Visual imagery elicits auditory imagery in other ways relevant to communication media. Calvert and associates, for example, found that “linguistic visual cues are sufficient to activate auditory cortex in normal hearing individuals in the absence of auditory speech sounds” (Calvert et al. 1997, p. 593). In the context of silent cinema, even as we watch actors soundlessly mouthing dialogue, a crossmodal mechanism performs a type of lip-reading. The phenomenon is robust not only for human speech but also for musical instruments and environmental sounds. Meyer and colleagues found the “subjective experience of sound, in the absence of auditory stimulation, was associated with content-specific activity in early auditory cortices” and when “subjects viewed sound-implying, but silent, visual stimuli, activity in auditory cortex differentiated among sounds related to various animals, musical instruments and objects” thus supporting the claim that “early sensory cortex activity reflects perceptual experience, rather than sensory stimulation alone” (Meyer, K et al. 2010, p. 667). In effect, exposure to silent moving pictures generates a complementary mental soundtrack of speech, music and environmental sounds. Conversely, as the BRECVEMA framework (Juslin 2013b, p. 242) reveals in Chapter 5, audition exerts a similar but contra effect in the induction of visual imagery115.

The impoverished imagery of silent cinema prompts our perceptual system for completion116, and the stylistics of silent movies reflect this impulse. In the years before the transition to synchronous sound-film117, filmmakers developed a range of techniques to exploit the kind of mechanisms for crossmodal interaction described by Calvert (Calvert et al. 1997) and Meyer (Meyer, K et al. 2010). These visual design strategies catalysed a soundtrack in the audience’s imagination in creative response to the technological limitations of non-synchronous film. Melinda Szalóky (Szalóky 2002) addresses this stylistic development in her analysis of the ‘visual acoustics’ of F.W. Murnau’s Sunrise: A Song of Two Humans118 (Murnau 1927). Visual acoustics precipitate auditory imagery. Sounds “become visualized and accentuated by the distinctive use of a pictorial cinematic device (typically close-ups, camera movements, editing, and the gestures of actors)” (Szalóky 2002, p. 127, emphasis in original). The

89 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. enhancement of visual composition, montage and rhythm to kindle sonic imagery suggests the leveraging of a covert multimodality. Moreover, it suggests cinematic imagery to be innately synaesthetic. Taken together, this research illuminates the defects of a purely visual definition of cinema: human beings ‘hear’ silent moving pictures, both voluntarily and involuntarily.

Crossmodal interaction as mechanisms for cinematic design If the ocularcentric theorising of silent cinema is muddied by contemporary neuroscientific findings the situation becomes radically more complex when exploring synchronous sound- film. Above I claimed cinema to be intrinsically multimodal (where sound wields the capacity to modify visual perception, and vice versa). Through professional practice, I have come to understand that careful crafting of sound has the capacity to influence the phenomenal quality of visual objects, and these qualities imbue affective meaning. This affective meaning, moreover, is not necessarily dictated by narrative context.

Sound may communicate the material nature and texture of an object or alter its perceived mass, speed or momentum. It may suggest physical contact between objects when no contact occurred. Sound may be manipulated to cause the visual image to appear crisper or brighter. Sound may concretise or dematerialise a visual image. It may supply or suppress a visual image’s apparent tactile or gustatory qualities. Sound also possesses the capacity to influence the affective perception of persons, objects and events. Even more remarkable is the capacity of sound to intensify the ‘energy’ of a scene, even if the visual image is ‘slow’ or ‘empty’119. This last phenomenon – the energy of a scene – is as subtle as it is significant for it refers, in movie-making terms, to an audience’s engaged attention. Energy is just another way of describing an audience’s affective engagement with the unfolding of events. Moreover, the design of sound may steer the eye’s passage across a screen, or draw the eye to some objects but disregard others. I consider this capacity of sound to steer visual attention and shape temporal sequence to be a sense-making function and, by extension, provide sound design with the capacity to narrate.

A tacit knowledge of crossmodal interaction appears to provide a basis for the practice of sound design, and it behoves an embodied cognition approach to cinematic sound to identify and understand how these fundamental mechanisms are used as vectors for design.

90 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Crossmodal interaction and integration are highly promiscuous processes which yield a rich array of phenomena. Each of these phenomena are potential mechanisms for cinematic design. Modality dominance is one such class. The visual dominance of audition in the spatial domain is, of course, extremely well-known. Good illustrations of visual dominance over audition are to be found in the ventriloquism effect (Bertelson 1999; Bertelson & Aschersleben 1998; Thurlow & Jack 1973) (whereby vision captures auditory spatial location) and the speech illusion of the McGurk effect (McGurk & MacDonald 1976) (whereby vision modifies speech perception). The utility of the ventriloquism effect reveals itself in the context of theatrical exhibition where the spatial capture of sound by vision creates the perception that an actor’s speech issues not from the location of the loudspeakers but from the facial image projected upon a screen. Lesser known, however, is the capacity of audition to wield dominance over vision.

In a review of literature, Shams, Kamitani & Shimojo (Shams, Kamitani & Shimojo 2004) note multiple ways in which audition influences visual perception, particularly in the temporal domain. Sound, for example, influences visual aspects of duration, rate, and event order (Shams, Kamitani & Shimojo 2004, p.27). Sound influences the enhancement or degradation of visual temporal resolution (Scheier, Nijwahan & Shimojo 1999). It may alter visual event perception such as in the sound-induced flash effect, an illusion whereby a single flash of light presented in coincidence with two successive and rapid auditory beeps is perceived as two flashes (Shams, Kamitani & Shimojo 2002). Particularly significant to the design of cinematic sound is temporal ventriloquism, a phenomenon whereby sound improves the fluency of visual processing. For example, a visual event may be perceived as occurring earlier when preceded by a sound but occurring later when followed by a sound (Aschersleben & Bertelson 2003; Bertelson & Aschersleben 2003; Fendrich & Corballis 2001; Keetels & Vroomen 2011; Vroomen & de Gelder 2004). Similarly, a coincident sound with a rapid onset improves the detection of a visual object in a rapidly changing sequence (McDonald, Teder-Salejarvi & Hillyard 2000; Vroomen & de Gelder 2000). This auditory influence over visual salience is known as perceptual freezing, or ‘pip-and-pop’ because a “synchronous [aural] pip makes the visual object pop out from its complex environment” (Van der Burg et al. 2008, p. 1053). Arrighi, Marini & Burr explored the ‘pip-and-pop’ phenomenon in relation to the perception of tap-dance and found “meaningful auditory information enhances perception of visual biological motion”, and, more generally, the “synchrony between auditory and visual events cannot only improve visual performance … but also affect visual perception

91 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. phenomenologically” (Arrighi, Marini & Burr 2009, p. 5). Surprisingly, non-informative sound also improves visual motion perception. Kim, Peters & Shams found that “concurrent auditory motion stimuli improve accuracy even when they do not provide any useful information for the task” (Kim, Peters & Shams 2012, p. 6). Concurrent sound appears to narrow the window of uncertainty concerning the timing of visual information (Arrighi, Marini & Burr 2009) suggesting crossmodal interaction provides a tighter lock on external reality.

The pip-and-pop phenomenon has great utility as a vector for cinematic design because it facilitates the isolation, identification and enhancement of visual objects, particularly in action sequences where rapidly changing visual fields are encountered120. An example of this phenomenon is found in the T-rex battle sequence in King Kong (Jackson 2005, between 01:40:55 and 01:42:55 (hh:mm:ss) approximately). Not only is this a sequence of rapid visual montage where each frame rolls vertiginously through XYZ axes, but Ann Darrow (Naomi Watts) is thrown through the frame at whirlwind speed. Indeed, it is quite difficult to keep track of Darrow’s spatial location throughout the sequence. However, with the aid of a sudden vocalisation or sound effect the figure of Darrow (or the dangerous end of a dinosaur) is momentarily arrested and surges to our attention.

Figures 3.1–3.8. Sound design perceptually freezes moments from a stream of rapidly presented visual information. Screenshots from King Kong (2005).

Similarly, beat entrainment for the purpose of perceptual freezing is a marked design strategy in the teaser for The Girl with the Dragon Tattoo (Fincher 2011). Musical beats typically motivate edits in picture montage but the teaser’s unrelenting snare does something quite different121. It acts to isolate, freeze and identify a visual moment by falling either just before or just after a picture edit. These beats arrest moments of change in facial expression, in bodily gesture, or vehicular movement, and surge to attention as moments of significance122. These moments provide rhythmic anchorage to affectively structure the mediawork and thus create nodes upon which to base an associative network. Moreover, the musical beat accelerates the processing of visual flow, allowing information to cascade rapidly over the audience without necessarily coalescing into narrative. The perceptual freezing of the pip- and-pop effect is a powerful tool for cinematic design especially when linked to mood induction through rhythmic entrainment.

Figures 3.9 and 3.10. Aural beats induce rhythmic entrainment and perceptually freeze affective moments. Screen shots from The Girl with the Dragon Tattoo teaser (2011).

Sound influences visual motion perception Sound not only exerts influence over visual temporal perception but also visual motion. For example, a static visual image may be sonically manipulated to appear to move not only laterally (Freeman & Driver 2008; Getzmann 2007; Hidaka et al. 2009; Kafaligonul & Stoner 2010; Teramoto, Hidaka & Sugita 2010) but also vertically (TeramotoManaka, et al. 2010); visual motion may be biased toward an auditory stimulus’ motion (Meyer, GF & Wuerger 2001); and sound may modulate visual momentum (TeramotoHidaka, et al. 2010). Another perceptual illusion significant to cinematic design is the stream/bounce effect whereby two disks moving toward each other in a mute condition appear to stream through one another, but when a brief sound is introduced at the moment of visual coincidence the discs are perceived to bounce (Sekuler, Sekuler & Lau 1997; Shimojo, Shinsuke et al. 2001; Watanabe & Shimojo 2001). Although these perceptual effects are automatic and innate, some audiovisual motion effects are modulated by cultural knowledge. For example, cultural norms are known to influence the co-perception of changes in pitch and visual motion (Maeda, Kanai & Shimojo 2004; Sadaghiani, Maier & Noppeney 2009; Takeshima & Gyoba 2013). These effects are extensible to film music. In the Western tradition, Hedger et al. found the “repeated exposure to rising or falling musical scales shifted participants’ sensitivity to visual motion in the opposite direction” and proposed the “subjective interpretation of monotonic pitch change as motion may have a perceptual foundation” (2013, p. 1039). Hedger and colleagues also noted the influence of cultural variation in metaphoric mapping in that the “perception of metaphoric motion in music may rely on the same perceptual mechanisms by which we process visual motion, [but] this does not necessarily mean that the auditory frequency–vertical space metaphor is innate” (Hedger et al. 2013, p. 1043). Exemplifying

94 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. these concepts in the context of cinematic sound is the theatrical trailer Gravity “I've Got You”123 (Cuarón 2013a) which illustrates how motion information is extracted from music and crossmodally transferred to visual imagery. Similarly, with regard to computer-generated animation, sound effects have been found to significantly increase the perceived smoothness of rendered motion (Mastoropoulou, G et al. 2005) as well as guide visual attention (Mastoropoulou, Georgia et al. 2005). Taken together, these kind of crossmodal effects suggest that common neural substrates are involved in the processing of auditory and visual motion (Hidaka et al. 2011) which, in turn, suggests the senses are much more interwoven than ocularcentrism admits.

Sound influences phenomenal qualities of visual perception Multimodal interactions influence the phenomenal quality of visual objects in a number of ways. For instance, auditory cues have been shown to enhance visual intensity (brightness)124 (Frassinetti, Bolognini & Làdavas 2002; Odgaard, Arieh & Marks 2003; Stein, BE et al. 1996), loudness to correlate with height such that louder is perceived as higher (Eitan, Schupak & Marks 2008), and pitch to correlate not only with the visual perception of size such that lower pitch is larger but rising pitch is perceptually linked with decreasing distance (Eitan 2013; Eitan et al. 2014). Crossmodal correspondences are also known to tie pitch to haptic qualities such as viscosity (Asad, Spiller & Jonas 2016) and sound transients to roughness (Etzi et al. 2016). Moreover, these kind of correspondences are used extensively in metaphorical mapping during music perception (Eitan & Rothschild 2011). Crossmodal correspondences also exist between sound and taste (Crisinel & Spence 2010; Knöferle & Spence 2012; Knöferle et al. 2015). Sound (in this case, music) has been found to influence the perceived qualities of wine (North 2012), beer (Carvalho et al. 2016), chocolate (Reinoso Carvalho et al. 2017), and the spiciness of food (Wang, Q, Keller & Spence 2017). These findings, it should be noted, have been made in relation to music and may not be extensible to everyday environmental sounds, but the recentness of this work indicates crossmodal research is still in its infancy.

Sound concretises a visual image A conspicuous characteristic of synchronous sound-film is that sound appears to concretise, to make more real, the visual image. Stephen Prince makes the important distinction between the reality effect (understood within apparatus theory as a function of ideology which lends naturalness to cinema125) and cinematic realism (understood as a “correspondence-based

95 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. model of cinematic representation” (1996, p. 31)). The perceptual reality of cinematic imagery arises, in part, through the mechanism of multimodal integration. The function of multimodal integration (in particular, audiovisual fusion) in concretising a visual image and the subsequent production of cinematic space is well illustrated by contrasting Louis Lumière’s La Sortie de l'Usine Lumière à Lyon (1895) with William K.L. Dickson’s Dickson Experimental Sound Film126 (1894/1895). La Sortie is a strong example of visual space created by the movement of the workers as they stride toward camera and fan to the left and right of the factory doors to exit the frame. The visual space of the Dickson Experimental Sound Film, however, is limited to an extremely shallow plane. Although La Sortie models deep visual space, this space is foreshortened as a consequence of its silence. Mindful of Gorky’s spectral experience of silent moving pictures, the workers in Lyon, even as they exit the factory, are somehow already dead, walking through the ‘Kingdom of Shadows’. In contrast, the visual plane of Dickson’s Experimental Sound Film is inordinately shallow but its cinematic image possesses, paradoxically, a greater sense of presence as a consequence of the synchronisation of aural and visual imagery.

The realism induced by audiovisual fusion is not zero-sum. There exists a dynamic spectrum of interaction. For example, the design of auditory features may be selectively rendered or omitted, and features may appear, disappear or qualitatively metamorphose over time. It is interesting to note that selectively sculpted sound often expresses a highly subjective point-of- audition as exemplified in the blitzkrieg sequence in Come and See (Klimov 1985), the Omaha Beach landing in Saving Private Ryan (Spielberg 1998), the Tokyo night club scene in Babel (Inárritu 2006), and the VHS sequence in The Ring (Verbinski 2002). Each of these sequences are strong examples of selective aural rendering which acts to concretise or dematerialise a visual image127.

Figure 3.11. Deep visual space but impoverished production of presence. Screenshot from La Sortie de l'Usine Lumière à Lyon (Lumière, 1895).

Figure 3.12. Enhanced production of presence through multimodal fusion. Screenshot from Dickson Experimental Sound Film (Dickson, 1894/95).

97 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Another curious difference between the moving visual image and its multimodal cousin is the effect of sound upon the experience of narrative time. Silent moving pictures appear temporally adrift, seeming to occur in the past or a possible future, and thus make ambiguous the reality-status of the imagery. Consider, for example, the phenomenon of music videos created for the purpose of movie promotion. Such music videos typically re-edit the visual content of the original movie, stripping away its dialogue, sound effects and music score but overlay a featured song. A celebrated music video of this genre is the theme song for the movie Titanic (Cameron 1997), My Heart Will Go On (Woodruff 1997). At play is something more than a ‘reality effect’. The visual imagery appears temporally ambiguous and narratively unanchored, and the visual world of the movie, as it is contained within the music video, is experienced at an alienated distance128.

Theorists of early cinema believed the concretising effect of sound threatened cinema’s status as art because it violated the concept of medium specificity129. Medium specificity, Noël Carroll reminds us, is the idea that “each art form, in virtue of its medium, has its own exclusive domain of development” (Carroll, Nl 1985, p. 5). For theorists intent upon proving cinema to be a fledgling artform that exclusive domain was visuality, and the ‘talking film’ was an abomination which violated medium purity. Rudolf Arnheim, in particular, drew heavily upon the concept of medium specificity in arguing his resistance to synchronous sound-film, claiming it to be aesthetically inferior to ‘silent’ cinema’130 (1938, pp. 215-216). It is instructive in the corrosive resilience of ocularcentrism within Western philosophy that a gestalt psychologist such as Arnheim should disregard its central tenet of holism in favour of a purely visual account of cinema. Indeed, it is only at the close of the 20th century that Arnheim came to recant his position, recognising the multimodal nature of cinema:

I see now that there is no such thing as a work limited to a single medium, if ‘medium’ is understood in a broader sense. Any narrative film is a representational work, which has to interact with the conditions of reality. Reality insists on certain properties and excludes others. These ‘natural’ conditions must harmonize with the formal qualities of the film medium. Even abstract film must be about something, and this function mandates its own formal choices.

The film medium, as I recognize now, profits from a freedom, a breathing space, that I could not afford to consider when I fought for the autonomy of the cinema. It is free to

98 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. use sound or no sound, color or no color, a limited frame or an endless space; it can exploit depth or use the virtues of the flat plane.

This freedom puts the film more closely in the company of the other performing arts, such as the theater, the dance, music, or pantomime. But film can do things they cannot do. I would call film a technically enhanced version of performance. (Arnheim, Rudolph 1999, p. 561, emphasis is mine)

From a cognitive neuroscience perspective, the human perceptual system seeks verification of the reality-status of an event through crossmodal confirmation: Is the imagery emanating from an external or internal reality? A key function of multimodal integration, therefore, is the production of presence.

Our perceptual systems were forged by evolution to provide us with veridical impressions of objects and events in the world. Our different senses give us different information about the same things. The information from different modes is compared, and this cross-checking leads to confirmation or denial of the nature and position, and even the potential consequences, of objects and events in the world. That’s how we as biological creatures interface with our environment. And (this is the important point for us as film scholars) we see and hear movies the same way we see and hear the real world. (Anderson, JD 1993, p. 32)

The impulse of cinematic media, as I shall argue in Chapter 7, is the impulse toward perceptual and psychological immersion, toward both the feeling of ‘being there’ and of understanding the mind of a mediated character. The impulse of cinema is toward synchronous audiovisual imagery (Gunning 2001). It facilitates multimodal fusion and perceptual immersion, and piggybacks upon ancient and automatic mechanisms for the production of presence. Presence, “lies at the center of all mediated experiences, from reading a novel (Gerrig 1993) to riding an immersive virtual reality (VR) simulator (Heeter 1992)” and the “desire to overcome the limit of human sensory channels through the use of technological devices is the major impetus for the development of media and reality- simulation technologies” (Lee 2004b, p. 27). Multimodal fusion is a basic building block of cinematic sound design not only for creating presence but, in the case of its cinematic manipulation, for enabling multimodal metaphor131.

Sound influences the affective perception of persons, objects and events The dimension of affect extends and complicates the spatiotemporal matrix of crossmodal interaction. Affect is instrumental in intermodally binding human voice and face132 (Pourtois et al. 2000), and has been shown to occur independently of higher cognitive attentional resources (Vroomen, Driver & de Gelder 2001). This mandatory binding is bidirectional such that affect detected in audition influences visual perception and vice versa (de Gelder & Vroomen 2000). Not only does affect facilitate integration and encoding to memory but it speeds its subsequent retrieval (Versace & Rose 2007). Similarly, music exerts affective influence over visual perception. Musical emotion, for example, crossmodally transfers to bias facial perception (Jomori et al. 2012; Logeswaran & Bhattacharya 2009), and musically induced mood has been shown to influence the affective ratings of still pictures133 (Baumgartner, Esslen & Jäncke 2006; Baumgartner et al. 2006). Indeed, the strongest affective responses are not “evoked by presenting emotional pictures alone (at least in most cases), but only by simultaneous presentation of congruent emotional musical excerpts” (Baumgartner, Esslen & Jäncke 2006, p. 42). Emotion elicited by music score also crossmodally transfers to moving pictures (Ellis & Simons 2005; Hanser & Mark 2013; Parke, Chew & Kyriakakis 2007; Tan, S-L, Spackman & Bezdek 2007), and has been shown to strongly influence audience response to movie characters, particularly with regard to “likability and the certainty of knowing the character’s thoughts, which are antecedents of empathetic concern and emphatic accuracy” (Hoeckner et al. 2011, p. 146). This kind of affective transfer from score to character has been labelled the “auditory Kuleshov effect” (Baranowski & Hecht 2016). Music fundamentally alters the way we see the world (Jolij & Meurs 2011).

Historically, the majority of studies in crossmodal emotion perception focus upon voice and face and, to a lesser extent, music. What is lacking, from the perspective of sound design theory, are studies which go beyond social communication to explore the crossmodal transfer of affect from naturalistic environmental sounds. Research by Gerdes and associates (2014; 2013) represents some of the very few studies into the crossmodal effects of emotional sounds across a range of sound categories. Specifically, Gerdes et al. found that complex sounds modulate the affective processing of still photographs. This research reveals that not only do “interactions of visual and auditory emotional information are not limited to social

100 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. communication but can extend to much broader contexts including human, animal, and environmental cues” (Gerdes, Wieser & Alpers 2014, p. 1).

There are several aspects of this research which are particularly significant to the theory of cinematic sound I am developing. The first relates to the finding that “emotional sounds may unspecifically increase sensory sensitivity or selective attention … to all incoming visual stimuli” (Gerdes, Wieser & Alpers 2014, p. 5). This resonates with James A. Russell’s concept of core affect, the significance of which is discussed in Chapters 4 and 5. However, as Gerdes and her associates acknowledge, research into the crossmodal transfer of affect from environmental sound to moving pictures is in its infancy, and there is currently no model to explain the integration of multisensory emotional information (Gerdes, Wieser & Alpers 2014, p. 10).

Another significant aspect of this research pertains to the nature of emotion perception itself. In the experiments by Baumgartner and associates, affective pictures were found to “evoke a more cognitive mode of emotion perception, whereas congruent presentations of emotional visual and musical stimuli rather automatically evoke strong emotional feelings and experiences” (Baumgartner et al. 2006). These findings indicate there are two distinct modes of emotion perception at work. The first recognises emotional features (suggesting top-down influences), and the latter experiences emotional arousal (suggesting bottom-up influences). These different modes of emotion perception are reflected within my embodied simulation theory of cinematic sound as the entwined interaction between the bottom-up processes which drive perceptual design and the top-down processes shaping narrative design (detailed in Chapter 7).

Sound influences visual attention One of the more contentious assumptions I have drawn from professional experience is the claim that sound may exert influence over visual attention. The ‘pip-and-pop’ effect (Kim, Peters & Shams 2012; Van der Burg et al. 2008), noted above, enhances the detection of visual objects through perceptual freezing, and music has been shown to optimise visual processing through beat entrainment (Escoffier, Herrmann & Schirmer 2015; Escoffier, Sheng & Schirmer 2010), but what evidence indicates sound wields the capacity to steer visual attention?

101 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Research by Coutrot and associates (Coutrot et al. 2012) represents some of the earliest studies of the effect of soundtrack on visual gaze. Their research reveals a complex relationship between picture montage and visual gaze, and the influence of sound upon that gaze. Firstly, it was established that an audience undergoes at least four phases in a visual- only condition when viewing complex moving picture scenes. The first phase ranges from frame 1 to 3 (~120 ms) where gaze remains in the same screen location as the previous shot. The second phase ranges from frame 4 to 10 (~240 ms) where gaze returns to the centre of the screen. The third phase ranges from frame 11 to 25 (~500 ms) where gaze begins to explore the visual scene. Lastly, the fourth phase ranges from frame 25 and onwards where gaze oscillates around a stationary value but driven in a bottom-up manner as new salient visual regions appear. These phases of visual gaze provide a strong model for the kind of visual attention generated by the montage of motion pictures. In the audiovisual condition134, sound did not have a discernible effect upon visual gaze during the first three phases, but from frame 25 onwards sound appears to facilitate a number of effects; namely, “eye positions of participants are less dispersed and tend to go more away from the screen center, with larger saccades” and, more significantly, “observers do not look at the same locations when videos are seen with or without sound” (Coutrot et al. 2012, p. 9).

Research of this kind begins to answer important questions about the kinds of interaction visual and auditory attention undergo during exposure to audiovisual scenes, but does not speak to the details of their interplay nor address the individual effects of speech, music and sound effects upon visual attention. Rassell, Redmond, Robinson, Stadler, Verhagen & Pink (Rassell et al. 2016), exploring to what extent vision follows narrative-based sound cues, build upon the rapidly expanding literature on audience eye-scanning by establishing that “when sound is present in a scene, it helps direct and focus where and for how long viewers look” (Rassell et al. 2016, p. 161).

Rassell and colleagues analysed discrete subsequences from two quite different genres; namely, the beach landing sequence from the live-action feature Saving Private Ryan (Spielberg 1998) and a chase sequence from the animated feature Monsters, Inc. (Docter, Silverman & Unkrich 2001). These subsequences135 contain “complex sound design, moments of perceptual shock, spatial and temporal shifts in sound, and heightened sonic agency” (Rassell et al. 2016, p. 141). Specifically, the research revealed that in an audiovisual context136 the audience “move their eyes significantly less, remaining longer engaged in a

102 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. smaller number of looks” (2016, p. 149). Sound also has the effect of tightening the visually scanned area around the protagonist leading to the suggestion that the audiovisual context elicits a more narrative mode of engagement (2016, p. 154). Conversely, in the visual-only condition, visual gaze notably spreads out and wanders, and “viewers spent more time … looking at the edges of the screen and not just in the central area” (Rassell et al. 2016, p. 151). Sound, therefore, appears to anchor the location of visual gaze, especially to face, eyes and mouth, but also to the environment when so cued (2016, pp. 153-154). Notably, some visual gestures in the audiovisual context are invisible but gain audience attention in the visual-only context. In terms of sound aesthetics, sound design techniques which foster explicitly subjective point-of-audition extend the duration of gaze fixation and are capable of generating deep internal focalisation (Rassell et al. 2016, pp. 145-146 and 160). Additionally, sound appears to promote and orchestrate a form of “attentional synchrony” across audience members (Rassell et al. 2016, p. 149) which I consider to be a soft form of emotional and cognitive contagion. Taken together, these studies suggest, as previously observed by Wooley & March, “audio enhances a viewer’s ability to efficiently navigate a dynamic scene, and without it, the viewer is left making more inefficient, and/or different visual allocations” (Wooley & March 2014, p. 190).

Although this research goes some way toward establishing a true model137 of audio-visual saliency it lacks detailed analysis of the effects of timbre and dynamics of individual soundtrack elements upon visual attention (such as voice versus music versus other forms of sound design). Interesting in this regard is the recent research from Song, Pellerin & Granjon which suggests different sound types affect visual gaze differently, and that social stimuli (such as human speech and music) have a greater effect upon visual attention than non-social sounds so that the “human voice drives participants to move their eyes towards the sound source” (Song, Pellerin & Granjon 2013, p. 11). This effect is explicitly illustrated in the multi-screen composition of Timecode (Figgis 2000) where synchronous speech seems to ‘activate’ the screen sector to which the audience will (mostly) attend. Other concrete examples of sound design steering visual attention through complex or rapidly changing visual displays, such as in genres like action-adventure or thriller, is the T-rex battle sequence in King Kong (Jackson 2005). In this sequence sound provides guidance to visual attention through a highly dynamic series of threats and opportunities. However, this steering function is not solely a quirk of blockbusters but also occurs in comparatively tranquil movies as well. In Mon Oncle (Tati 1958, at approximately 18:30 minutes), when a stylish guest pays a visit,

103 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. the harsh sound of a boy’s shoes scraping on a doormat attracts our visual attention even though the boy is in deep background and a highly animated conversation between the son’s mother and her guest is underway in the foreground138. The tacit understanding and manipulation of these kind of attentional interactions is the bread-and-butter of the professional sound designer.

Figure 3.13. Synchronous speech activates the screen sector to which an audience will (mostly) attend. Screenshot from Timecode (Figgis, 2000).

Figure 3.14. Shoes scraping on a doormat attracts visual attention from foreground to background action. Screenshot from Mon Oncle (Tati, 1958).

Auditory dominance before visual dominance Conventionally, human beings are considered to be visually dominant creatures (Colavita 1974; Posner, Nissen & Klein 1976). Visual dominance is the attentional bias toward visual information in an audiovisual stimulus. While the findings which support visual dominance have been consistently replicated these studies focus exclusively upon adults. Visual dominance has not been found in neonates or young children. On the contrary, a body of research reveals a strong auditory dominance in infants (Lewkowicz 1988a, 1988b). Research by Vladimir Sloutsky (2010; 2003; 2008), Amanda Napolitano (2006; 2003, 2004) and Christopher Robinson (2004, 2007) reveals human beings to be aurally dominant until as late as four years of age, while a recent study by Nava & Pavani (2013) found visual dominance begins to emerge at approximately 9–10 years of age and becoming fully dominant by 11–12 years of age. How is it we are born aurally dominant and then effect a switch to visual dominance139? What does auditory dominance afford us in developmental terms during these crucial first years of life? One explanation is that audition is already active in utero while visual cognition, for all intents and purposes, is dormant. Time is required after birth for vision to mature the necessary competencies to rival audition. Regarding the function of

105 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. auditory dominance, Sloutsky (2010) indicates that perceptual grouping of word and non- word sounds in the categorisation of objects provides an infant with the basic cognitive architecture for labelling and the subsequent formation of abstract concepts. The work of Sloutsky, Napolitano and Robinson suggests much conceptual groundwork is laid down in the first years of life, and it is upon this architecture that visual cognition piggybacks. It may be that auditory dominance provides conceptual ‘bootstrapping’ for vision.

In summary, sound modifies visual perception (and vice versa) in many subtle and complex ways. Indeed, the senses exert polysemous influence upon each other according to rules which are the subject of ongoing research140. The rules which govern multimodal interaction are presumed to have ecological origins (Shams, Kamitani & Shimojo 2004, p. 28), and an important companion to the above review of audiovisual interaction is that by Shams & Kim (Shams & Kim 2010) which examines proprioceptive and tactile modalities. Taken together, abundant evidence overwhelmingly affirms human beings to be profoundly multimodal creatures. Vision rarely, if ever, works in isolation from the other senses and this suggests our perceptual system seeks verification of the reality-status of an event through crossmodal confirmation141.

Sound, synaesthesia and affect Synaesthesia was once believed to occur in only a limited percentage of the population but recent research suggests that synaesthetic processes are not only active in normal individuals but “appear to play a crucial (if unacknowledged) role in the multisensory integration of auditory and visual information” (Parise & Spence 2009, p. e5664). Synaesthesia, as Jamie Ward defines it, is the “involuntary perceptual experience in one sensory modality that is normally associated with another perceptual modality or with amodal cognitive processing” (Ward, J 2004, p. 761). Moreover, there is an important link between crossmodal interaction, synaesthesia and affect. Research in vocal and facial expression indicates “[i]ntermodal binding between affective information that is seen as well as heard triggers a mandatory process of audiovisual integration” (Pourtois et al. 2000, p. 1329). While this finding is highly significant for human communication, it also suggests the existence of a general autonomous mechanism for parsing sensory information for affective content. The concept of affective synaesthesia (Collier 1996) labels this cross-culturally robust capacity to map emotions onto even simple auditory and visual stimuli.

106 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Turning to the crossmodal transfer of emotion from audition to vision, Logeswaran & Bhattacharya have shown that “listening to musical excerpts, albeit short, could significantly influence the subsequent explicit evaluation of visual emotional stimuli” and that “musical priming could also influence implicit visual emotional processes” (2009, p. 133). In a similar vein, Baumgartner and associates note that “music can markedly enhance the emotional experience evoked by affective pictures” (2006, p. 34) and contrasts this with the observation that “[silent] emotional pictures evoke a more cognitive mode of emotion perception” (2006, p. 151). However, it is Richard Cytowic’s insights into synaesthesia and affect which have the most significant bearing upon a theory of cinematic imagery:

I believe no general theory of synesthesia is possible without accounting for the characteristic affective state that accompanies synesthetic experience. The experience of affect is understood to depend on the limbic brain… [T]he human brain’s neocortical expanse does not configure itself as a replacement for the emotional limbic brain. The neocortical mantle is not the highest rung on the ladder of cognition, completely suppressing everything below it, but evolved as a detour on the ladder, interposed between brainstem and limbic brain.

The limbic brain in an advanced mammal like the human retains its role for suppressing automatic pre-existing biases in favour of newer alternatives to express themselves, and is a locus where value, purpose, and memory are calculated. This organ for calculating valence, to use a more neutral term, could have had either of two fates. It could had [sic] been suppressed by neocortex, which is to say that one’s evaluations of value and bias are now to be replaced by higher calculations, a so- called better organ for determining meaning and purpose. That seems to be the quintessential belief of the intellectual. He or she thinks that the goal of thought is to suppress limbic function – to suppress emotion. But the neocortex does not calculate emotion or valence. It just calculates. (Cytowic 2002, p. 10, emphasis in original)

People who make their choices emotionally, as we say, are more human than those who make them rationally exactly for this reason. You should permit your intellect only to inform your choices, not to override fundamentally emotional ones. This is a roundabout and romantic way of saying that synaesthesia can perhaps be looked on as

107 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. a short-hand way of calculating valence and salience, of attaching meaning to things. (Cytowic 2002, p. 11, emphasis in original)

Cytowic locates the capacity for human meaning-making not in the analytics of rational logic but in our emotionality. For Cytowic, synaesthesia provides a window into the operation of automatic crossmodal processes where emotional valence and salience act as a wellspring for meaning. Insights such as these align with Mark Johnson’s concept of embodied meaning (2007b, pp. 10-11). Synaesthetic effects arising from multimodal integration provide vectors for perceptual design, and inform a theory of cinematic imagery because they indicate how the aesthetic manipulation of cinematic imagery can produce embodied meaning142.

In summary, a central tenet of Film Studies is the primacy of vision, a consequence of which is an unchallenged ocularcentrism. However, the conception of a purely unimodal visual cinema cannot stand in the face of amassing empirical evidence which reveals the radical extent to which the human senses are interconnected. “[S]ensory modalities are not separate modalities” as neurobiologists Shinsuke Shimojo and Ladan Shams plainly observe (Shimojo, Shinsuke & Shams 2001, p. 505), and this should dissuade us from speaking of cinema as a ‘visual medium’. Knowledge of such deep and interpenetrating crossmodal effects must inform an account of cinematic media with ocularcentrism abandoned in favour of a more comprehensive and heterogeneous approach.

3.5 Sound design and embodied meaning The working definition of sound design, elaborated upon in Chapter 2, describes a process by which many sound fragments are created, selected, organised and blended into a perceptually unified and coherent auditory image. Distinctive features of contemporary sound design include a playful recombination of sonic fragments, and a heightened manipulation of auditory spatialisation, temporal resolution and timbre.

A consequence of the migration of sound design from magnetic film to digital audio workstations (DAWs) is a radically enhanced ability to manipulate aural imagery. DAWs offer significant advantages over traditional magnetic film-sound. For example, DAWs permit the editing of sound at the subframe level so that a sonic fragment may be accurately isolated, extracted, re-aligned, layered and blended with other fragments by increments approaching 1

108 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. millisecond143. These fragments may be further shaped through software-based processing, such as equalisation, reverb and echo, gating, phasing and flanging, time compression/expansion, pitch-shifting, harmonic manipulation and other specialised psychoacoustic effects. What these techniques yield are immensely detailed sonic textures. DAWs also afford the innovative recombination of those auditory fragments in relation to a visual image through the manipulation of audiovisual fusion. Audiovisual fusion, discussed above, is a form of multimodal integration. Its mechanism is impenetrable to cognition, and occurs within a temporal window of between −30 ms and +170 ms144 (van Wassenhove, Grant & Poeppel 2007, p. 598). Audiovisual fusion also produces a feeling of presence, and it acts as the neuropsychological driver behind one of the most fundamental and powerful building blocks of cinematic sound design: audiovisual metaphor145.

Kathrin Fahlenbrach (2002, 2005b, 2005a, 2008) has written extensively on audiovisual metaphor and its affective dimensions. Audiovisual metaphor acts as a form of sense-making by synaesthetically mapping qualities from a source onto a target domain. From audiovisual metaphor it is barely a hop to conceptual metaphor, and from there – as Fauconnier and Turner might put it – to ‘running the blend’ and ‘seeing as-if’ (2002). To illustrate how the re- association of aural and visual imagery is capable of synthesising multimodal and conceptual metaphor consider how the sound design of the Recognizer (a type of futuristic police airship) in TRON: Legacy (Kosinski 2010) is comprised of electrical crackles, high-performance turbine whines, wind wisps, and a tearing of air that borders on distortion146. The aural qualities of the Recognizer, when mapped onto its visual image, not only concretises its reality-status but defines the vehicle as heavy, powerful and capable of extreme force. In addition, embedded within this sound effect is a chunkiness147 reminiscent of 8-bit ‘old skool’ computer games, and so a cultural allusion is made between TRON the sequel and TRON the original (Lisberger 1982). Incorporated into a single sound effect are the environmental cues of heat, force and power, the cultural cues which signal power and control over social space, and associations to an earlier generation of computer-based entertainment which inform the audience of this flying machine’s heritage. It is this essentially synaesthetic form of audiovisual re-association that is the engine of cinematic meaning-making because it generates affectively laden multimodal metaphor and facilitates conceptual blending148.

Digital technology has also facilitated other conspicuous features of contemporary sound design, such as an increased density in auditory cuing149, and the capacity to generate

109 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. surround soundfields with exceptionally wide frequency response and high dynamic range. A consequence of this is the production of auditory imagery with greater definition and presence. However, the most significant aspect of contemporary cinematic sound is its heightened manipulation of timbre.

In some ways, the contemporary focus upon timbre may be considered a rejoinder to the

Futurist call-to-arms made by Luigi Russolo when he urged the abandonment of 19th century musical instrumentation for the embrace of an “art of noises” (1913, p. 6). The DAW, as a device for generating and assembling noise, is example par excellence of Russolo’s intonarumori. However, the shift toward a heightened manipulation of timbre in post-classical sound design suggests something more significant is at play150. Timbre characterises not only sonic identity but pre-attentive categorisation associated with expression and emotional imagery (Goydke et al. 2004; Meyer, M, Baumann & Jancke 2006).

From a historical perspective, the praxis of sound design exhibits a tendency toward techniques and technologies with greater and greater immersive characteristics in the production of an aesthetic experience151. Following Ed Tan’s framework for understanding the function of art and entertainment as “an episode of emotions in response to an ongoing guided imagination” (Tan, ESH 2008, p. 28), I propose the primary concern of cinematic sound design is the production and structuring of affective content152. In short, sound design is a form of emotional design. Further, I propose affective cinematic content may be achieved through two design strategies: what I term perceptual design and narrative design. For the purposes of this thesis, I shall concentrate upon the function of perceptual design and its relationship to Wojciehowski & Gallese’s concept of Feeling of Body because, firstly, much critical analysis has already been focussed historically upon narrative at the expense of the perceptual and, secondly, in recognising Wojciehowski & Gallese’s claim that Feeling of Body provides the embodied basis for narrative (2011, pp. 7-8) it is possible that understanding perceptual design may be the more essential153.

3.6 Perceptual design and narrative design Perceptual design and narrative design are forms of immersion achieved by different routes154. Perceptual immersion inducts core affect, mood and feeling-states within the body of the audience through (mostly) pre-attentional, nonconscious mechanisms which process

110 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. multimodal information while narrative immersion structures emotion and meta-emotion through (mostly) attentional, conscious mechanisms. Perceptual immersion is characterised as ‘bottom-up’ processing while narrative immersion is characterised as ‘top-down’ processing155. A rich target for the analysis of these two forms of design is Sucker Punch (Snyder 2011c), a spectacular action fantasy movie that received almost universal critical disdain. As part of a pre-release promotional campaign, a teaser (Snyder 2011a) and trailer (Snyder 2011b) were produced for online distribution. Discussed in detail in section 7.1.1, these two mediaworks advertise the same movie but illustrate very clearly the contrasting strategies of perceptual and narrative design156.

From a functional perspective, perceptual immersion is the simulation of physical experience (Bastiaansen, Thioux & Keysers 2009; Gazzola, Aziz-Zadeh & Keysers 2006; Keysers & Gazzola 2009; Keysers et al. 2003) while narrative immersion is the simulation of social experience (Lee 2004b, 2004a; Lombard & Ditton 1997; Mar & Oatley 2008). As a consequence of the mechanism of embodied simulation, narrative immersion arises from perceptual immersion. The primary function of these forms of immersion is to both produce and consume affective content. From these definitions it is possible to claim the output of perceptual immersion is Feeling of Body while the output of narrative immersion is Theory of Mind.

Theorising narrative as a kind of virtual machine for the production of affect is not new. Ed Tan (1996) explicitly describes narrative cinema as an ‘emotion machine’, and David Miall (1988, 2011) and Patrick Colm Hogan (2011) extend classical narratology toward an ‘affective narratology’. For Hogan, “story structures are fundamentally shaped and oriented by our emotion systems” (2011, p. 1). However, in adopting Nico Frijda’s construal of emotion as a type of appraisal (1986, 1988), Tan’s conceptualisation of narrative cinema uses a distinctly ‘cognitive’ understanding of emotion which does not capture the underlying function of embodied processes I wish to analyse.

Is it possible to experience cinematic meaning before the emergence of narrative? Patrick Keating (2006) notes that theoretical models of cinematic narrative and spectacle are typically grouped into three categories:

111 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. … a Classical model, which argues that a certain type of narrative operates as a dominant in relation to various subordinate systems; an Alternation model, which argues that the dominance of narrative alternates with the dominance of other systems; and an Affective model, which argues that linear narrative is itself subordinate to a more important goal, the production of emotion. (Keating 2006, p. 4)

Keating rejects all three models and instead suggests a ‘co-operative’ model where

[n]arrative and other attractions can work together to produce an intensified emotional response. We can call this the Co-operation model, since the model explains how narrative and attractions can support each other. (Keating 2006, p. 4)

Keating’s observations regarding the affective raison d'être of both narrative and ‘attractions’ resonate with Richard Walsh’s pronouncement of the general failure of musicology’s turn to narratology in the latter part of the 20th century (2011). Walsh observes that narratives are fundamentally representational while music is not, but

[w]hat narrative and music share, then, is a relation to the articulation of temporal affect; while music elaborates upon this in a systemic, experiential mode at the cost of intentional specificity, narrative elaborates and objectifies its sequential structure at the cost of the immediacy of systemic immersion. (Walsh 2011, p. 66)

In both narrative and music Walsh recognises common somatic, affective and social bases although they are “neither directly analogous nor unrelated”, and exist as “forms rooted in a common set of attributes, which substantially constrain and determine the nature of narrative understanding and communication” (2011, p. 67). Walsh notes these bases are also the features of “protonarrative” (2011, p. 56).

I adopt a somewhat different construal of proto-narrative while respecting Walsh’s observations. Proto-narrative, as I define it, is a template comprised of affectively linked embodied schema. It acts as a cinematic interface between perceptual and narrative immersion because it transduces sensorimotor patterns into abstract concepts as a function of ‘liberated’ embodied simulation (Wojciehowski & Gallese 2011, p 2). From this perspective, perceptual design is a strategy which manipulates what Lawrence Barsalou calls perceptual

112 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. symbols (1999) and what Gallese & Lakoff call cogs (2005), both of which theorise the reuse of neural resources of the sensorimotor system in the production of abstract conceptual knowledge157. In addition, Charland expands upon Barsalou’s account of perceptual symbols by proposing the inclusion of affect because “emotion can be argued to form a distinct symbol processing system of its own” (1999, p. 613). Within an embodied cognition account of sound design, the core affect and mood generated by perceptual design acts to form structures of proto-narrative.

3.7 Proto-narrative and the cinematic Above I described cinema as an affective spatio-temporal system. What makes cinema cinematic is expressively crafted movement through multimodal space and time. Cinematic content is affective content. It is generated in response to audiovisual stimuli as a consequence of the activation of sensorimotor schemas: processes that happen before and below conscious thought. I believe this is the type of content Tom Gunning describes when he speaks of the cinema-of-attraction’s possibility for “an intense interaction between an astonished spectator and the cinematic smack of the instant, the flicker of presence and absence” (1993, p. 11). By this definition, the smallest cinematic unit is a burst of presence, and it is these bursts, when sequenced together, which form the basis of what I call proto- narrative. Useful here is the concept of interoception, the body’s own composite internal physiological image which provides basis for the self as a feeling and sentient entity (Craig 2002, 2003; Wiens 2005). Interoception is also proposed as the neural basis for time perception and the emergence of awareness because it links emotionally salient beats across time (Craig 2009).

Cinematic proto-narrative, then, refers to an affective linking of the body’s experience of mediated space through time. Because it proceeds by a wide, fast and networked logic of association I suggest proto-narrative is analogous to Ned Block’s concept of phenomenal consciousness (P-consciousness). Block defines the content of P-consciousness as the “experiential properties of sensations, feelings and perceptions” (1995, p. 230) and contrasts it to access consciousness (A-consciousness) which is only available “for use in reasoning and rationally guiding speech and action” (1995, p. 227). Using Block’s typology of consciousness, the term perceptual design addresses P-consciousness while narrative design addresses A-consciousness. A similar correspondence is to be found, respectively, between

113 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. perceptual design and narrative design and António Damásio’s concepts of core consciousness and extended consciousness (Damásio, AR 1998). In Chapter 6 the relationship between embodiment and consciousness is discussed in depth.

The logic of proto-narrative is analogous to the logic of P-consciousness because it is not, strictly speaking, narrative in nature158. Proto-narrative is the flow of affect which occurs before and below the logic of consciousness and the narrative self. As in the relationship between P-consciousness and conscious thought, proto-narrative is the embodied basis for narrative proper. Proto-narrative is the flow of body-based image schema bound together into an experiential episode by an unfolding affective logic.

Cinema, as Tom Gunning (1986, 1993) reminds us, is neither intrinsically nor necessarily narrative; there is always that which eludes narrative’s chain of cause and effect159. It is cinema’s ability to produce bursts of presence that is exceedingly relevant to a theory of cinematic sound, and it roars to the fore in post-classical cinema as “audiences are encouraged not just to listen to sounds but to ‘feel’ them” so they may “hear the unhearable” (Sergi 1998, p. 162).

3.8 Feeling of Body Feeling of Body (FoB) is a concept developed by Hannah Wojciehowski and Vittorio Gallese (Wojciehowski & Gallese 2011) to explain the visceral experience of reading. It emerges from a conjoining of neuroscience and literary studies which they describe as an embodied narratology160. Noting the current uptake of cognitivism within the humanities, Wojciehowski & Gallese counter the consequent emphasis upon Theory of Mind by arguing for the primacy of Feeling of Body. Wojciehowski & Gallese differentiate Feeling of Body from Theory of Mind by contrasting the former’s capacity to bring forth intercorporeity with the latter’s capacity to bring forth intersubjectivity, and note “intersubjectivity should be viewed first and foremost as intercorporiety” (Wojciehowski & Gallese 2011, p. 7). However, because both Feeling of Body and Theory of Mind arise from embodiment, no dualism should be inferred.

Wojciehowski & Gallese describe Feeling of Body as a ‘liberated’ form of embodied simulation, and is

114 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. quite different from standard accounts of the Simulation Theory of mind-reading. Embodied simulation is a mandatory, pre-rational, non-introspective process – that is, a physical, and not simply ‘mental’ experience of the mind, emotions, lived experiences and motor intentions of other people … [E]mbodied simulation creates internal non-linguistic ‘representations’ of the body-states associated with actions, emotions, and sensations within the observer, as if he or she were performing a similar action or experiencing a similar emotion or sensation. This is what the Feeling of Body (FoB) amounts to. By means of the neural format we share with other human beings, and, to an extent, with some animals, as well, we can map others’ actions onto our own motor system, as well as others’ emotions and sensations onto our own viscero-motor and somatosensory systems … Consequently, the FoB is not to be uniquely conceived of as a mere sensing of how our body reacts to external stimuli. It is a bodily way of making sense of our social world. (Wojciehowski & Gallese 2011, p. 14)

Feeling of Body underpins Theory of Mind by foregrounding the role of the body in sense- making and social interaction. In this way, Feeling of Body puts into play the ramifications of Mark Johnson’s theory of embodied meaning which proposes all abstract conceptual thinking and imagination have basis in our biology (Johnson, M 2007b, p. 10). Embodied simulation, as I argue in Chapter 5, provides an engine for media aesthetics which is derived from the body’s capacity for the simulation of another’s emotional and mental state, even that of a fictional other.

------this page intentionally blank ------

4.1 Embodied cognition The humanities and social sciences were not alone in experiencing a turn to the corporeal and affective in the last decades of 20th century. A similar turn manifested within the cognitive sciences. By at least the late 1980s, arising from a dissatisfaction with traditional understandings of human behaviour and thought, an interest in how cognition might be embodied began to ferment161. However, the conceptualisation of embodiment which developed contrasts with that examined in Chapter 2, and to understand the relevance of embodied cognition to Film Studies it is important to appreciate its departure from ‘standard’ cognitive science.

Cognitive science is the study of the mind, and the structure and function of the mechanisms which give rise to it. Far from unified, cognitive science is a confederacy of psychology, linguistics, artificial intelligence, neuroscience, philosophy and anthropology, and is marked by an interdisciplinarity deemed essential to solving its multivariate target problems. Thus far cognitive science has undergone three developmental waves: computational, emergent, and enactive162.

The first wave – computationalism – is also known simply as ‘cognitivism’, and is often referred to as “standard cognitive science” (Shapiro 2011, p. 7). Examples of cognitivism often include the work of Jerry Fodor, Noam Chomsky, and Zenon Pylyshyn163. The classical account presents cognition as disembodied, and so suffers the ‘Cartesian anxiety’ of mind– body dualism (Colombetti 2010, pp. 145-146)164. Computationalism is also profoundly non- affective because it affords no significance to the role and function of mood, feeling or emotion. While computationalism’s manipulation of symbols might produce a convincing account of cognition it has proven limited in its real-world application. This failure has driven some researchers, particularly in robotics and artificial intelligence, to seek new explanatory paradigms. These new paradigms are emergent and enactive cognition, and they directly challenge the Cartesian representationalism at the core of standard cognitive science165.

117 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Emergent cognition, also known as ‘connectionism’, is cognitive science’s second wave166. Connectionism emphasises “the idea that many cognitive tasks (such as vision and memory) seem to be best handled by systems made up of many simple components, which, when connected by the appropriate rules, gives rise to global behaviour corresponding to the desired task” (Varela, Thompson & Rosch 1991, p. 8). Researchers working with theories of emergence and connectionism include Michael Arbib, David Marr, Ulric Neisser, and Daniel Dennett.

Enactive cognition is the third and most recent wave167. Enactivism builds upon the insight from connectionism that understands mental and behavioural phenomena to form via emergent processes that interconnect neural networks. It emphasises cognition to be “not the representation of a pregiven world by a pregiven mind but is rather the enactment of a world and a mind” (Varela, Thompson & Rosch 1991, p. 9). A foundational text in establishing the sub-discipline is The embodied mind: cognitive science and human experience (1991) by Francisco Varela, Evan Thompson and Eleanor Rosch168. Although Varela, Thompson and Rosch initially labelled the approach ‘enactive’, Thompson has since recognised the necessity of tuning the term toward ‘embodied’ cognition’ (Campbell 2012, at approximately 00:09:55- 00:10:20 (hh:mm:ss))169

Embodied cognition (EC) is less a theory than a research programme unified by its commitment to “elevate the importance of the body in the explanation of various cognitive abilities” (Shapiro 2007, p. 340)170. Specifically, Margaret Wilson (2002) identifies six central claims associated with the EC paradigm:

The emerging viewpoint of embodied cognition holds that cognitive processes are deeply rooted in the body’s interactions with the world. This position actually houses a number of distinct claims, some of which are more controversial than others. … (1) cognition is situated; (2) cognition is time-pressured; (3) we off-load cognitive work onto the environment; (4) the environment is part of the cognitive system; (5) cognition is for action; (6) off-line cognition is body based. Of these, the first three and the fifth appear to be at least partially true, and their usefulness is best evaluated in terms of the range of their applicability. The fourth claim, I argue, is deeply problematic. The sixth claim has received the least attention in the literature on

118 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. embodied cognition, but it may in fact be the best documented and most powerful of the six claims. (Wilson, M 2002, p. 625)

Wilson’s six claims are of direct relevance to a theory of cinematic media. Claims 1 and 2 – “cognition is situated” and “cognition is time-pressured” – recognise cognition to be embedded in an environment which evolves through time and is therefore subject to real-time pressures of interaction with that environment. Cognition arises through the sensorimotor loops formed by the body in interaction with the environment within which it is embedded. This is to say, cognition arises through enaction and sustains claim 5, “cognition is for action”. These claims are significant to a theory of cinematic media because they foreground the fundamental importance of bodies and objects moving and interacting through space and time. Additionally, claims 3 and 4 – “we off-load cognitive work onto the environment” and “the environment is part of the cognitive system” – suggest unique affordances may be ascribed to cinema when considered as an ‘environment’, albeit a virtual one171. Cinematic media brings about cognition through the activation of the same kind of sensorimotor schemas with which we directly engage the real world. Cinema, from this perspective, appears to function as a ‘cognitive tool’172.

Lastly, there is claim 6 – “off-line cognition is body based”. Bluntly put, cognition is ‘online’ when dealing with the immediate environment but is ‘offline’ when deliberating, remembering or engaging in imaginative or counterfactual thought. Claim 6 is crucial to a theory of cinematic media because it indicates how sensorimotor representations provide the necessary resources for mental activity. Wilson notes that “[e]ven when decoupled from the environment, the activity of the mind is grounded in mechanisms that evolved for interaction with the environment – that is, mechanisms of sensory processing and motor control” (2002, p. 626). In making a distinction between ‘online’ and ‘offline’ cognition, Wilson portrays online cognition as the mind “operating to serve the needs of a body interacting with a real- world situation” (2002, p. 635), but offline cognition is an entirely different matter. In contrast, it includes

any cognitive activities in which sensory and motor resources are brought to bear on mental tasks whose referents are distant in time and space or are altogether imaginary. These include symbolic off-loading, where external resources are used to assist in the mental representation and manipulation of things that are not present, as well as purely

119 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. internal uses of sensorimotor representations, in the form of mental simulations. In these cases, rather than the mind operating to serve the body, we find the body (or its control systems) serving the mind. (Wilson, M 2002, p. 635)

Offline cognition, manifesting as imagination and counterfactual thought, reuses the same sensorimotor schema with which we feel and think in the real world173. Thomas Suddendorf and Michael Corballis (2009; 1997, 2007) call this faculty mental time travel, and it appears to be most vital in human beings for the purpose of remembering the past to plan the future, or for entertaining entirely fictional or counterfactual situations to generate novel information for use in future online scenarios. Cinematic media are highly parasitic upon these underlying embodied processes, and may be understood as an instantiation of a heavily guided form of offline cognition.

In summary, embodied cognition is a programme of research which radically departs from traditional cognitive science by reframing cognition as ‘embedded action’. By centralising the body and its sensorimotor processes, EC challenges conventional computational models and rejects Cartesian dualism by conceiving of the mind as inherently embodied: the body–mind. Embodied cognition also suggests how consciousness, imagination and narrative comprehension may proceed via the reuse of sensorimotor patterns as perceptual symbols (Barsalou 1999, 2003b; Glenberg & Robertson 2000; Zwaan 1999) and embodied schema (Johnson, M 2005; Schnall 2014). All these processes have clear relevance to the design and reception of cinematic media. However, for embodied cognition to be useful as a paradigm it must also be capable of explaining the function of affect.

4.2 Emotion science What is an emotion, and how and why does it come about? Which concepts and terminology are relevant to the study of emotion? If debating the nature and function of cognition is contentious the study of emotion is no more tumultuous. The limitations of this chapter do not permit a thorough examination of the controversies in emotion research but a summary of several key issues and insights will put us in good stead in understanding the relationship between embodied cognition and affect, and the implications these insights have for a theory of cinematic media.

120 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Theories of emotion in Western philosophy date back at least as far as Hippocrates, Plato, and Aristotle174. Of these extant accounts, it is Aristotle’s with which we are most familiar. In Poetics (Aristotle c. 335 BCE) Aristotle not only proposed significant theories of narrative and drama but also a role for emotion in the function of the arts (particularly the ‘tragic’ emotions) as well as the mechanism of catharsis for purging negative emotions in service of a social and moral purpose. Aristotle, therefore, provides our earliest example of a media psychology. Important theories of emotion in Western philosophy are also to be found in the speculations of Descartes, Spinoza, Hobbes, and Hume. Crystallised in this intellectual history are the all-too-familiar binaries of mind-versus-body, reason-versus-emotion, and so on. These binaries imply a hierarchy of importance. Cartesian dualism, for example, elevates soul (reason) over body (emotion) to imply the purpose of the body is to act merely as a vehicle for the mind, and an imperfect vehicle at that. This view, of course, mirrors the dualism held by standard cognitive science that the mind is a kind of computational software that runs on the hardware of the body. In contrast, theories of embodied cognition vigorously refute the either/or dichotomies of these mind/body dualisms.

In the later part of the 19th century a scientific study of emotion began to emerge with Charles Darwin’s The expression of the emotions in man and animals (Darwin 1872) and William

James’ ‘What is an emotion?’ (James 1884). However, in the early 20th century the belief that emotion could be empirically studied fell out of favour with the rise of behaviourism, and would only return with the later waves in cognitive science175. Likewise, a cooling (if not outright suspicion) towards human emotion developed within the humanities. The literary critics William Wimsatt and Monroe Beardsley (1949, 1954) famously critiqued the relevance of a reader’s emotional response to a literary work as an ‘affective fallacy’. Wimsatt & Beardsley argued emotional response had no place in true criticism because emotions confused what the work is with what the work does. On Wimsatt & Beardsley’s account, considerations of emotion could only lead to a hopeless relativism. In essence, the affective fallacy invalidates the reader’s emotional response to art as irrational. The effect of Wimsatt & Beardsley’s bifurcation of emotion from rationality was to hobble subsequent decades of scholarly research in the aesthetic function of literature and art by skewing critical theory in favour of an avowedly cool logic176. Ironically, the kind of ‘cold’ logic which Wimsatt & Beardsley championed in literary theory is analogous to the kind of cognitivism against which contemporary theories of embodied cognition react.

121 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. A serious impediment to advancing a science of emotion has been a dizzying and imprecise taxonomy177. Indeed, as James A. Russell relates, a recurring theme in the literature is the issue of taxonomy and the confusion this imprecision breeds:

The key culprit in this mess is the concept of emotion, or affect, as it is now sometimes called. Emotion is too broad a class of events to be a single scientific category. As psychologists use the term, it includes the euphoria of winning an Olympic gold medal, a brief startle at an unexpected noise, unrelenting profound grief, the fleeting pleasant sensations from a warm breeze, cardiovascular changes in response to viewing a film, the stalking and murder of an innocent victim, lifelong love of an offspring, feeling chipper for no known reason, and interest in a news bulletin. … No one structure of description and assessment can do justice to this heterogeneous class of events without differentiating one type of event from another. (Russell & Barrett 1999, p. 805)

Affect, emotion, feeling and mood are terms with exceptionally fuzzy definitions and, to make matters worse, these terms are often used interchangeably178. For the sake of consistency and transparency, I shall follow the terminology and definitions used by Patrik Juslin and John Sloboda (2010, p.10). Juslin and Sloboda specify affect as the umbrella term under which all other valenced states are located, where valency is the simple ascription of being either positive or negative, pleasurable or displeasurable, attractive or repulsive, or being in a state of tension or release. Emotion is a brief and intense affective state focussed upon a specific object. Mood is a prolonged but low intensity affective state which is free-floating because it lacks an object. Feeling is the subjective experience of an emotion or a mood. Additionally, and most importantly, I make use of James A. Russell’s concept of core affect (2003, 2005, 2009; 1999).

Core affect requires explanation so that its full relevance to an embodied simulation account of cinematic media may be appreciated. Contrary to the commonsense assumption that an emotion is unitary, Russell considers emotion to be a multicomponent affective process (Russell 2009, p. 1260)179. Within this process, Russell proposes core affect to be a primitive, universal and irreducible atom from which the psychological construction of emotional experience proceeds. Core affect is “simple at a subjective level but complex at the biological level” (Russell 2003, p. 148). Induced subliminally, one may sense core affect only as a non-

122 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. reflective feeling of good or bad, listless or energetic, and its free-floating nature allows it to “emerge into consciousness as a mood in some circumstances but [for example] as liking for a beverage in other circumstances” (2003, p. 148). Core affect is significant to a theory of cinematic media because it describes an intrinsic and fundamental neurobiological mechanism by which emotions are constructed. These constructions may emerge into consciousness or just as likely remain below the threshold of awareness. Core affect has special significance within an embodied cognition approach to cinematic media because it “responds to virtual reality in art, imagination, fantasy, and entertainment” (Russell 2009, p. 1266)180.

4.2.1 Competing theories of emotion Emotion research is a battlefield of competing theories. The early somatic theories of emotion (exemplified by William James and Carl Lange) are counterposed by cognitive theories (exemplified by Richard Lazarus, George Mandler, Robert Solomon, and Nico Frijda) which, in turn, are counterposed by neo-somatic theories (exemplified by Robert Zajonc, John Cacioppo, António Damásio, and Joseph LeDoux). Moreover, perceptual theories of emotion (exemplified by James Laird and Jesse Prinz) hybridise somatic and cognitive theories. In stark contrast to the affective fallacy, contemporary emotion theory places affect at the centre of cognition and the processes of meaning-making. The sheer complexity of what might constitute a robust theoretical framework of emotion is daunting, and yet, as Chapter 5 will show, an understanding of the purpose and function of affect is central in the explication of an embodied simulation account of sound in cinematic media. To this end, I shall sketch the general landscape of theories of emotion using Bartch & Hübner’s emotional communication theory (Bartsch 2004, 2005; Bartsch & Hübner 2005) which shows how these competing theories are not as adversarial as might first appear but may be fitted together to produce an integrated framework181.

In developing a theory of emotional communication, Anne Bartsch and Susanne Hübner observe theories of affect may be fruitfully grouped into four types: i) neuroscience models, ii) appraisal theories, iii) prototype approaches, and iv) social constructionist theories. These four types, a précis of which may be found in Appendix A, are often positioned as competing explanations of what an emotion is, how it arises, and what function it performs. However, Bartsch & Hübner observe these theories may be reconciled by understanding each to be descriptions of affect at different levels of cognitive complexity; namely, innate stimulus response patterns, associative schemata, and symbolic concepts.

Bartsch & Hübner’s emotional communication theory is significant to an embodied cognition approach to cinematic media because it provides a model for integrating sub-personal neurobiological processes with the personal, social and cultural. It explicitly recognises “superordinate levels of emotional communication involve automatically communication on the more basic levels”, and, moreover, “emotional communication on a symbolic level will not work if it does not activate nonverbal emotion scripts, which, in turn, depends on the activation of emotional brain systems” (Bartsch & Hübner 2005, pp. 4-5). This is to say, the symbolic communication of emotion at the level of social construction fails to operate without the body’s affective neurobiology to support it whereas the reverse is not true: social construction of emotion is profoundly rooted in neurobiology. These observations resonate with Mark Johnson’s theory of embodied meaning (2007b, pp. 10-11) which argues even the highest levels of human abstract thinking have their basis in the lowest levels of the biological182.

To bind together their emotional communication model Bartsch & Hübner use the concept of ‘cognitive appraisal’183, and this brings us into contact with a major controversy within the field of emotion research: Is emotion an aspect of the cognition system, a separate system altogether, or is cognition an aspect of the emotion system? This controversy, with its overtones of Cartesianism, orbits between those who argue for the ‘primacy of affect’ and those who argue for the ‘primacy of cognition’. This controversy is known as the Zajonc- Lazarus debate, and has great bearing upon an embodied cognition approach to cinematic sound because what is central to the debate is an understanding of the embodied nature of affect.

4.2.2 ‘Primacy of affect’ versus ‘primacy of cognition’ In 1980 Robert Zajonc published ‘Feeling and thinking: preferences need no inferences’ (1980) which radically claimed “[a]ffective reactions can occur without extensive perceptual and cognitive encoding, are made with greater confidence than cognitive judgments, and can be made sooner” (1980, p. 151). Zajonc took issue with the dominant view within psychology that regarded emotion to arise from cognitive processing, and argued the reverse to be true: emotion arises prior to cognition, and functions independently of it184. Defence of the dominant view was mounted by Richard Lazarus (1982) and others185. The peak of the dispute is encapsulated in Zajonc’s claim for the ‘primacy of affect’ (1984) against which

124 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Lazarus responded with a counter-claim for the ‘primacy of cognition’ (1984). Although final statements were published by both Lazarus (1999) and Zajonc (2000) the dispute produced little consensus other than revealing much of the debate’s heat stemmed from misunderstandings rooted in different conceptualisations of cognition.

Zajonc’s work in noncognitive emotion did not, at that time, overturn the dominant paradigm of cognitive appraisal within emotion research but it did act as a flag around which a subsequent generation of researchers rallied186. These researchers have taken up questions concerning emotion–cognition interaction to develop progressively more sophisticated models of noncognitive, unconscious affective processing. Paula Niedenthal observes that Zajonc, who was a social psychologist, was “quietly waiting for indisputable evidence coming out of the new field of neuroscience” (2010, p. 315) to validate his concept of affective primacy, and supporting evidence did, in fact, appear in the form of the neo-somatic theories of emotion developed by Joseph LeDoux and António Damásio among others. The work of LeDoux is of particular significance to both the Zajonc-Lazarus debate and a theory of cinematic sound design. LeDoux (1986, 1989, 1991, 1995) has theorised extensively upon unconscious emotional processing particularly with regard to fear and audition. Specifically, LeDoux identifies two parallel neural pathways in the processing of emotional auditory stimuli. The first is a “quick-and-dirty” sub-cortical pathway (LeDoux & Phelps 1993, p. 159), and the second is a slower cortical pathway capable of representing the auditory stimulus with greater efficacy.

On LeDoux’s account, the sub-cortical pathway confers speed and evolutionary advantage (1993, p. 159). Although this pathway is susceptible to error, from an evolutionary perspective it is beneficial to be fast and mistaken than slow and deceased. The phylogenetically older ‘bottom-up’ processing of the sub-cortical pathway produces a rapid affectivity before and below the threshold of conscious awareness whereas the ‘top-down’ cortical processing brings all the power of conscious attention to the situation to suppress, augment or generally modulate the initial nonconscious affective response. In effect, we ‘feel’ the presence of (and respond bodily to) a potential threat before we consciously identify the source187. These pathways indicate two distinct modes of address available for cinematic sound design; one targeted to generating core affect and inducing mood, and the other focused on communicating overtly narrative information.

125 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. The evidence from contemporary neuroscience not only supports the argument that emotion may occur before and below conscious awareness but also the claim that emotion drives learning and decision-making. Research by Jeffrey Saver and António Damásio (1991), and developed further by Damásio, Daniel Tranel and Hanna Damásio (1996; 1991) and Antoine Bechara (2005; 1994; 2005), suggests affective content in the form of ‘somatic markers’ acts to frame cognition and guide decision-making by providing a sensitivity to future consequences188. Taken together, these findings lead Damásio (1994, 1999) to claim cognition to be relatively weak in relation to the robustness of the human emotion system, and supports Zajonc’s claim for affective primacy189.

In summary, the significance of the body in neo-somatic theories developed by Zajonc, LeDoux, and Damásio return us to the import of a theory of emotion proposed a century earlier by William James (1884, 1890). Neo-somatic theories privilege embodied affect in human interactions with natural and social environments because bodily sensations automatically and rapidly attribute meaning to objects, actions and scenarios. Cognition, rationality and consciousness are undeniably useful for survival but are not enough without the power of emotion. The emotional unconscious has been “found, lost, and regained” (Kihlstrom, Barnhardt & Tataryn 1992), and Zajonc’s concept of unconscious emotion as a biologically driven, bottom-up mechanism for generating meaning, as Chapter 5 will show, has great significance for the design of cinematic media.

4.3 Embodied cognition and affect Theories of embodied cognition demand enactive models of emotion but, as Wojciehowski & Gallese wryly note, “[c]lassic cognitivism, like a dead star, still emits light” (2011, p. 11). Standard emotion theory, tied as it is to classical cognitivism, has “lost the body” (Colombetti & Thompson 2008, p. 50) and remains bound to a disembodied conception of emotion. Cognition from a cognitivist perspective

is an abstract, intellectual process that is not influenced by the state of the body. Experiential aspects of emotion, when acknowledged, depend uniquely on such abstractly characterized cognitive states. The tendency is to either deny their bodily aspect or reduced it to a by-product. (Colombetti & Thompson 2008, p. 49)

126 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. In contrast, theories of embodied cognition propose affective processing to be grounded in sensorimotor systems, and claim “what individuals ‘know’ about emotion is represented by their capacity to simulate or partly reexperience emotion” (Niedenthal & Maringer 2009, p. 122; Winkielman, Niedenthal & Oberman 2008, p. 281)190. Colombetti and Thompson privilege a concept of sense-making based upon sensorimotor subjectivity (Colombetti 2010, 2013, 2014; Colombetti & Thompson 2008; Thompson, Evan 2005) when developing an enactive theory of emotion. Sense-making “is behaviour or conduct in relation to environmental significance and valence, which the organism itself enacts or brings forth on the basis of its autonomy” (Thompson, Evan & Stapleton 2009, p. 25). This is to say, sense- making is a bodily form of meaning that is both generated and consumed by an organism in interaction with its environment (Colombetti 2010, p. 148)191. Sense-making, then, is what Mark Johnson calls embodied meaning (Johnson, M 2007b, pp. 10-11) – affect arising from body–environment interaction – and is critical, as Chapters 5 and 7 will show, to an embodied simulation account of cinematic sound.

Motion and emotion The enactivist claim that cognition arises through bodily interaction with the environment is developed extensively in the work of Maxine Sheets-Johnstone (1999b, 1999a). For Sheets- Johnstone motion is fundamental because

[a]nimate forms are the starting point of biological evolution. They are where life begins. They are where animation begins. They are where concepts begin. They are where emotions are rooted, not in something that might be termed ‘mental life’. (Sheets-Johnstone 1999b, p. 275)

Sheets-Johnstone connects motion and emotion as embodied sense-making when she suggests one performs “thinking in movement” (Sheets-Johnstone 2010) because “[w]hat is kinetic is affective, or potentially affective; by the same qualitative measure, what is affective is kinetic, or potentially kinetic” (Sheets-Johnstone 1999b, p. 259). This proposal has special relevance for cinematic imagery which is first and foremost moving imagery. Within Film Studies, the interrelationship between motion and emotion is under-researched; however, Valentijn Visch and Ed Tan (Visch 2004; Visch & Tan 2007, 2008, 2009) have shown that general audiences are highly skilled at apprehending emotion and genre prototypes from abstract patterns of

127 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. movement. This suggests moving imagery supplies a rich source of affective content even before the overt comprehension of narrative.

Although the research of Visch & Tan concentrates exclusively upon the motion of visual imagery their findings may be extensible to auditory imagery. Recent work in ecological psychoacoustics suggests this to be the case. Research in auditory looming and proximity undertaken by John G. Neuhoff (2001, 2004) reveals how the motion of auditory imagery is also capable of eliciting strong affective responses. The empirical study of ecologically valid auditory-induced emotion has only recently been established under the rubric of ‘emoacoustics’ yet it is beginning to yield fruit concerning the interrelationship between sound, space, movement, presence and affect192.

4.4 Friction at the nexus of scientific and cultural theory In Chapter 1 I presupposed a typical humanities scholar may not be versed in cognitive science nor a cognitive scientist with the stuff of the humanities. The humanities and sciences have a thorny history of talking past each other: at best, not completely grasping the relevance of each to the other; at worst, a degeneration into the infamous ‘science wars’193. An explanation of the tension between cognitive science and the humanities on issues of biology and culture is beyond the scope of this thesis but the discord it provokes requires exploration, if only partial.

A persistent and anxious feature in the background of this thesis’ research is the question of the relationship between the humanities and sciences, and the degree to which it impedes or facilitates the exchange of ideas. C.P. Snow (1959) identified a ‘two cultures’ schism between the sciences and humanities over five decades ago and yet the fluidity in contemporary interdisciplinary research remains less than ideal. Such a bifurcation in human understanding manifests in many ways, both small and large. Sometimes it manifests in the guise of misunderstood or misapplied critical vocabulary and concepts. Sometimes it manifests as outright hostility. Given the methodology of this thesis is to draw upon cognitive science to address long-standing blindspots within Film Studies concerning the praxis of sound design it is pertinent to explore, albeit briefly, the manner in which cultural theory baulks at the relevance of contemporary cognitive science.

128 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. An illustration of cross-disciplinary friction and misunderstanding is Kathryn Geurts’ critique of António Damásio’s neuroscientific account of consciousness (Geurts 2005). Geurts critiques Damásio’s approach on the grounds of a “restricted focus on individual rather than intersubjective consciousness” (Geurts 2005, p. 165), and wishes to show that the “culture- bound quality of many of his [Damásio’s] claims seriously weakens his position” (Geurts 2005, p. 164). Although Geurts objections are vigorously presented they are framed in a constrained and somewhat misleading manner. For example, Geurts’ claim that Damásio’s approach forecloses intersubjectivity is wide of the mark for this research demonstrably establishes the function of a common bodily format to be essential for intersubjectivity, language and social cognition194, a position further corroborated by the work of Gallese and colleagues (2008; 2015; 2005). Intersubjectivity, as Chapter 5 shall show when detailing Gallese’s embodied simulation theory, is primarily intercorporal.

Geurts’ critique also misrepresents the nature of Damásio’s ‘movie-in-the-brain’ metaphor to drive home a cultural anthropological account of embodiment. The target texts of Geurts’ criticism are Damásio’s Descartes' error: emotion, reason, and the human brain (1994) and The feeling of what happens: body, emotion and the making of consciousness (1999). Both these works attempt to communicate neuroscientific ideas to a lay Western audience. It is not surprising, therefore, that Damásio should reach for readily intelligible metaphors such as a ‘movie-in-the-brain’. As Geurts rightly points out, this move replays Descartes’ use of the “technology of the day” to metaphorically explain consciousness (Geurts 2005, p. 169). However, a closer reading of Damásio’s cinematic metaphor reveals an emphasis upon the multimodal and embodied nature of emotion and cognition. As Damásio intimates, his metaphor is not entirely accurate, and he goes to some length to clarify the ‘movie-in-the- brain’ to be not merely aural and visual but multimodal (Damásio, AR 1999, p. 9). Indeed, in describing the kind of images which comprise his ‘movie-in-the-brain’, Damásio specifically defines an ‘image’ as “a mental pattern in any of the sense modalities” but also includes other forms of imagery such as interoception, the image of inner bodily awareness (Damásio, AR 1999, pp. 8-9). Damásio’s use of a cinematic metaphor indicates more the paucity of language with which to communicate novel scientific concepts and less an indication of a “Western bourgeois ideology” rooted in visual technologism.

Although I do not deny the claim that cultural knowledge and ideology play a role in shaping cinematic technique, I am not persuaded of the claim it is ‘culture all the way down’. As

129 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. illustrated in Chapter 2, Mary Ann Doane’s (1985) heavy reliance upon the concept of suture within apparatus theory to unmask the putatively ideological nature of sound editing and mixing lacks the broad explanatory appeal of a neurobiological account offered by Drew Leder (1990). Leder argues the everyday function of the human sensory and perceptual systems are innately transparent to consciousness. These automatic processes of self- concealment are what Leder terms ‘bodily disappearance’. Following Merleau-Ponty’s work in phenomenology, Leder recognises the distinction between various forms of bodily disappearance. One such form is focal disappearance (1990, pp. 26-27). When applied in a cinematic media context, focal disappearance stands in contrast to Doane’s use of the concept of ideological suture. Focal disappearance is the “self-effacement of bodily organs when they form the focal origin of a perceptual or actional field. An example is the invisibility of the eye within the visual field it generates” (Leder 1990, p. 26). If the perceptual design of cinematic imagery is taken as analogue of sensory perception, the phenomenon of focal disappearance suggests itself as a more elemental explanation for ‘invisible’ style.

Leder’s concept of bodily disappearance offers a non-ideological approach to the question of how and why cinematic technique is typically rendered ‘invisible’: cinematic technique piggybacks upon the body’s natural tendency for self-erasure (Johnson, M 2007b, pp. 4-7). Ironically, this self-erasure appears to be the basis for a latent Cartesianism; a sense that the mind and the body are apparently two distinct and separate things. The result of bodily disappearance, Mark Johnson relates, “is our sense that our thoughts, and even our feelings, go on somehow independent of our bodily processes” (Johnson, M 2007b, p.6).

What appears to be at stake for cultural theorists such as Geurts and Doane is the primacy of culture in the creation and structuring of knowledge. From such a perspective, the prospect of a consilience between the humanities and sciences represents an existential threat requiring a “firewall”195 to be built around the standard paradigm of social constructionism196. That the humanities may be subsumed under the biological sciences is a palpable fear. Martin Jay, for example, even as he argues it “cannot be culture all the way down” (2002, p. 276) when detailing the intellectual overreach of radical culturalism, baulks at the kind of consilience proposed by E.O. Wilson (1998). Consilience, Jay maintains, is “the hostile takeover of the humanities by the natural sciences” (2002, p. 276). Misleadingly, this debate is often framed in either/or terms which overlooks more sophisticated possibilities. For example, evolutionary psychologists John Tooby and Leda Cosmides (Tooby & Cosmides 1992), in respecting the

130 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. inherent unity of knowledge, propose a vertical integration model of biological and human sciences197, and Edward Slingerland and Mark Collard198 recognise an emerging second-wave of interdisciplinary consilience that is attempting to develop a shared framework for the sciences and humanities (Slingerland & Collard 2011a, p. 4).

What the above goes to illustrate are the dangers inherent in working at the interdisciplinary nexus of the sciences and humanities. In Chapter 1, I acknowledged the hazards involved in an interdisciplinary approach to a conceptual framework for cinematic sound. Undoubtedly, misappropriation of scientific concepts (which appears to have occurred during the ‘affective turn’ within the humanities) is a serious danger. Constantina Papoulias and Felicity Callard (2010), for example, are sceptical of ‘affect theory’ as it is typically construed within the social sciences because of its “cultural–political commitment” (Papoulias & Callard 2010, p. 48) which they consider to be a series of mis-steps in translating scientific knowledge199. Ruth Leys (2011) has similarly critiqued the misinterpretation of neuroscientific concepts by ‘affect theorists’ such as Brian Massumi (2002a), Nigel Thrift (2004; 2008) and William Connolly (2002). Conversely, cognitive science, which has been more interested historically in human universals than individual and cultural difference, has failed to systematically engage with the issues central to cultural theory. Critiques of cognitive science of this kind are merited, and the recently emerged fields of social and cultural neuroscience indicate attempts are being made by the scientific community to address this lack by developing an understanding of the neural mechanisms by which similarity and difference may be explained at individual, social and cultural levels200. What currently divides cognitive science and the humanities is a difference in level of description not a difference in kind, and it would be unwise to believe knowledge in one domain cannot inform the other.

4.5 Summary: embodied cognition, affect, and meaning In summary, this chapter has sketched a broad terrain of scientific enquiry into the relationship between the human body, emotion and cognition. Specifically, this chapter has outlined the nature of the embodied cognition research programme and its radical break from standard cognitive science. It has outlined the nature of contemporary empirical theories of cognition, and laid out the fundamental role of affect in enabling cognition and meaning- making. Familiarity with this complex knowledge is essential to a sophisticated modelling of cinematic media, and prepares us for the detailed explication of an embodied simulation

131 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. account of cinematic sound encountered in Chapter 5 and the theories of consciousness and narrativity encountered in Chapter 6.

Encountered throughout this chapter is the contention within the cognitive sciences as to the relationship between emotion and cognition. Is emotion a form of cognition (Frijda 1988; Lazarus 1982)? Is cognition a form of emotion (Panksepp 2005)? Are emotion and cognition distinct and separate systems (LeDoux 1989)? Or are emotion and cognition so massively interconnected as to render the question meaningless (Okon-Singer et al. 2015; Pessoa 2013)? Which is more true: the ‘primacy of cognition’ (Lazarus 1984) or the ‘primacy of affect’ (Zajonc 1984)? In the face of such intense debate, and for the sake of transparency as I lay out an embodied simulation account of cinematic sound, I am persuaded to adopt the view that human cognition arose as evolutionarily specialised processes from the body’s affective system. From this perspective, the purpose of cognition and conscious awareness is to generate unique and novel information to be used as feedback by the affective system to modify, amplify or constrain behaviour. This leads me to follow Zajonc’s claim for the ‘primacy of affect’ (Zajonc 1984) which presupposes powerful and flexible nonconscious affective processes operate before and below consciousness. Rational logic has an affective core, and it is affect which yields shape and heft to meaning and embeds the body in the world (both physical and social). Taken together, these insights strongly suggest embodied affect must be placed at the centre of a theory of cinematic media. What is necessary, however, is a specific instance of a theory of embodied cognition which can act as a unifying theoretical framework for the multiple kinds of bodily, intersubjective and empathic experiences provided by cinematic media. In the following chapters I propose that embodied simulation theory satisfies many of the demands required of such a theory and I demonstrate how it can enhance our understanding of the function of sound in successful cinematic design.

5.1 Embodied simulation theory Embodied simulation theory, developed by Vittorio Gallese (2005, 2007a, 2009, 2011a, 2013; 2011a), is situated within the research programme of embodied cognition. It sets forth a “neurobiologically plausible and theoretically unitary framework” (Gallese & Sinigaglia 2011a, p. 512) when proposing that “a common functional mechanism is at the basis of both body awareness and basic forms of social understanding” (Gallese 2005, p. 24).

A core claim of ES [embodied simulation] theory is that similar constraints apply both to those representations of one’s own actions, emotions or sensations involved in actually acting and experiencing and also to the corresponding representations involved in observing someone else performing a given action or experiencing a given emotion or sensation; and that the constraints are similar precisely because the representations have a common (bodily) format.

In sum, ES is the reuse of mental states and processes involving representations that have a bodily format. The nature and range of what can be achieved with ES is constrained by the bodily format of the representations involved.… [B]odily format determines how ES contributes to mind-reading. (Gallese & Sinigaglia 2011a, p. 515)

From a historical perspective, embodied simulation theory is an elaboration of Gallese’s own shared manifold of intersubjectivity hypothesis (2001, 2003a, 2003b) which proposes a subpersonal mechanism supports the emergence of intersubjectivity. Embodied simulation theory is related to Wolfgang Prinz’ work in perception–action coupling (Prinz, W 1984) and common coding theory (Hommel et al. 2001; Prinz, W 1990, 1997, 2003) which argues perceptual and motor representation share a common neuronal encoding. This common encoding implies an equivalence between executing, observing and imagining a goal-oriented action. It also implies perception–action coding is accessible to an observer because it facilitates the retrieval of the intentions of the observed action and in doing so provides the basis for Theory of Mind and social interaction. Prinz’ common coding theory directly links perception and action. It rejects the classical ‘sandwich’ model of cognition which privileges

133 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. cognition as the supreme mediator between perception and action201. Instead, cognition arises from embodied action. Common coding, furthermore, resuscitates a previously abandoned line of inquiry which is traceable to William James’ concept of ideomotor response (1890), and anticipates the discovery of trimodal neurons, neurons “sensitive to the meaning of actions” (Di Pellegrino et al. 1992, p. 179).

Embodied simulation theory, supported by empirical research on the trimodal neuron system, extends Prinz’ account. Trimodal neurons “respond to motor, visual, and auditory stimulation, such as when an action is performed, observed, heard or read about” (Le Bel, Pineda & Sharma 2009, p. 299). The trimodal neuron system is also referred to as the ‘mirror neuron system’ (MNS) because of its ability to mirror-match observed action to produce motor resonance and simulation of the observed within the observer. A broad range of empirical studies indicate the MNS to be implicated in the facilitation of learning, language acquisition, mind-reading and social cognition202. Taken together, the evidence provided by these studies suggests embodied simulation is a neurobiological foundation for the social capacity to understand others as intentional agents because the “same neural structures that are involved in processing and controlling executed actions, felt sensations and emotions are also active when the same actions, sensations and emotions are to be detected in others” (Gallese 2003b, p. 171).

Embodied simulation is also claimed to be a mechanism for coding peripersonal space in egocentric terms (Gallese 2005, 2007b; Gallese & Sinigaglia 2011a). Peripersonal space is a spatial zone which is anchored to specific parts of the body, such as hands and mouth, to create a ‘motor space’ within which a repertoire of potential motor schemas may be simulated and executed (Gallese 2005, pp. 25-26). The function of motor space is to map objects and environments for affordances203.

Vision, sound and action are parts of an integrated system; the sight of an object at a given location, or the sound it produces, automatically triggers a “plan” for a specific action directed toward that location. What is a “plan” to act? It is a simulated potential action. (Gallese 2005, p. 27)

Embodied simulation theory is also related to Lawrence Barsalou’s perceptual symbol systems theory (Barsalou 1999, 2003a, 2005, 2008; Barsalou et al. 2003; Barsalou, Solomon

134 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. & Wu 1999), George Lakoff and Mark Johnson’s conceptual metaphor theory (Lakoff 1987; Lakoff & Johnson 1980b, 1980a), and Gilles Fauconnier and Mark Turner’s mental space theory (Fauconnier 1985) and conceptual blending theory (Fauconnier & Turner 2002; Turner & Fauconnier 1995). Each of these theories take the human body acting in space, and the sensorimotor schemas produced, as the basis for abstract thought.

In summary, embodied simulation theory describes universal, innate and nonconscious mechanisms which mediate “our capacity to share the meaning of actions, intentions, feelings, and emotions with others, thus grounding our identification with and connectedness to others” (Gallese 2009, p. 524). Bodies, motion, emotion and meaning are inextricably interwoven.

5.2 Function of audition within embodied simulation What does embodied simulation theory offer a theory of cinematic sound? Cinematic imagery, as I argued in Chapter 3, is multimodal imagery, moving imagery. Central to embodied simulation is the function of the MNS and its capacity to integrate auditory, visual and motor systems to produce embodied schemas204. Audition is at the heart of embodied meaning because “audiovisual mirror neurons code abstract contents – the meaning of actions” (Kohler et al. 2002, p. 846). MNS function and embodied simulation mechanisms provide a biological basis for the recognition and understanding of intention and affective content contained within auditory events (Gazzola, Aziz-Zadeh & Keysers 2006; Keysers et al. 2003; Kohler et al. 2002). Moreover, because “individuals who scored higher on an empathy scale activated this [auditory mirror] system more strongly” (Gazzola, Aziz-Zadeh & Keysers 2006, p. 1824), it may be that a relationship is to be found between embodied 205 simulation and the 19th century German concept of Einfühlung , or ‘feeling-into’. Taken together, embodied simulation mechanisms and MNS function suggest themselves to be key to a theory of cinematic sound design206.

Embodied simulation theory provides a unifying framework for cinematic practices because it describes how the body–mind’s own building blocks – variously known as image schemas (Johnson, M 1987; Lakoff 1987), cogs (Gallese & Lakoff 2005), perceptual symbols (Barsalou 1999) – produce conceptual knowledge207. An example of the cinematic deployment of these kinds of building blocks is audiovisual metaphor208. Audiovisual metaphor is a fundamental tool for cinematic design. It arises through manipulation of the

135 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. cognitively impenetrable and subpersonal processes of multimodal integration which produce audiovisual fusion209. Audiovisual metaphor provides the basis for conceptual blending and “seeing as-if” (Fauconnier & Turner 2002). “[I]magining is a form of simulation”, Gallese and Lakoff contend, “a mental simulation of action or perception, using many of the same neurons as actually acting or perceiving” (2005, pp. 456-457, emphasis in original). Embodied simulation is an engine for the production of cinematic imagery.

In the cinematic, we experience the mediated world as if through another body or, to use Wojciehowski & Gallese’s term, a “Feeling of Body” which is enabled by a “liberated” form of embodied simulation (Wojciehowski & Gallese 2011, p. 2). This ‘feeling of body’ is brought forth by the technological production of presence which exploits the tight fit with which our sensory and perceptual systems evolved to interface with the real world. Cinema recruits the same sensorimotor mechanisms we use to understand the real world: no special media-specific mechanisms are required. During the practice of sound design, aural presence is produced for the purpose of inducting a bodily experience of affect and, consequently, thoughts, for as Keith Oatley observes “emotions configure our cognitive systems” (2009, p. 208). The emotional power of cinematic imagery may be encapsulated in the claim that embodied simulation produces a “direct form of emotion understanding” (Gallese 2005, p. 37), a claim which resonates with Hans Gumbrecht’s analysis of ‘presence effects’ and hermeneutics which suggests cinematic sound is capable of providing access to the world in a way that interpretative meaning cannot (2004, pp. 138-141).

5.3 Embodied simulation and cinematic media

5.3.1 Embodied simulation theory and art Embodied simulation (ES) has been examined thus far within this thesis in the context of social cognition arising through real-world interaction and only covertly extended to encounters with mediated imagery. How legitimate is the extension of ES theory to art and entertainment? Although in its infancy, a growing body of work has begun to apply ES theory to the study of music (Koelsch et al. 2006; Koelsch & Siebel 2005; Menon & Levitin 2005; Molnar-Szakacs & Overy 2006), dance (Calvo-Merino et al. 2008), theatre (Blair 2008; Johnson, L, Sutton & Tribble 2014; McConachie, Bruce A. 2008), literature (Wojciehowski & Gallese 2011), film210 (Gallese & Guerra 2012a, 2014; Heimann et al. 2014; Ward, MS

136 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 2015), as well as to painting and sculpture (Freedberg & Gallese 2007; Gallese & Freedberg 2007) and art generally (Di Dio & Gallese 2009; Gallese 2011c; Gallese & Di Dio 2012).

In the context of literary theory, Gallese and Hannah Wojciehowski (Gallese 2011b, 2011c; Wojciehowski & Gallese 2011) propose the mechanism of embodied simulation may be applied to both the real and virtual worlds and, when experienced through art and entertainment, is experienced in a “liberated” form:

The aesthetic experience of art works, more than a suspension of disbelief, can be thus interpreted as a sort of “liberated embodied simulation.” When reading a novel, looking at a visual art work, or attending a theatrical play or a movie, our embodied simulation becomes liberated, that is, it is freed from the burden of modeling our actual presence in daily life. … Through an immersive state in which our attention is focused on the narrated virtual world, we can fully deploy our simulative resources, letting our defensive guard against daily reality slip for a while. (Wojciehowski & Gallese 2011, p. 17, emphasis in original)

Additionally, the kind of liberated ES which arises in narrated virtual worlds may also be deployed to describe the affective features which are physically encoded into the artwork itself.

The artwork becomes the mediator of the sensory-motor and emotional resonance that establishes between the artist and the observer, thus allowing beholders to feel the artwork in an embodied manner. Liberated embodied simulation hence provides a potentially unified level of description of both artistʼs and beholdersʼ relation with the art work. (Gallese 2011c, p. 65)

Embodied simulation, therefore, is capable of describing at least two forms of response to art: response to the representational content of the artwork, and response to the materiality of the artwork itself. These responses correlate with Ed Tan’s concept of representation emotions (R-emotions, emotions arising from a mediated world or imagined persons, objects and events) and artifact emotions (A-emotions, aesthetic emotions arising from experiencing the artwork as an artefact) (Tan, ESH 1996, pp. 81-84; 2000, p. 118).

137 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 5.3.2 Feeling of Body before Theory of Mind Feeling of Body is a ‘liberated’ form of embodied simulation (Wojciehowski & Gallese 2011). The intercorporeity of Feeling of Body provides the basis for – yet occurs before and below – the intersubjectivity of Theory of Mind (Wojciehowski & Gallese 2011, p. 7), leading Wojciehowski & Gallese to observe that

empathy may be conceived as the outcome of our natural tendency to experience interpersonal relations first and foremost at the implicit level of intercorporeity. Our interpersonal relations – both in daily life as well as with fictional characters – are marked by our bodily involvement (the FoB) with the actions, emotions and sensations acted and expressed by others. (Wojciehowski & Gallese 2011, p. 14)

Feeling of Body labels and foregrounds the role of the body in sense-making and social interaction. As a ‘liberated’ mechanism of embodied simulation, FoB provides an engine for media production and reception derived from the body’s innate capacity for the simulation of another’s emotional and mental state, even that of a fictional other211. Mediated experience, therefore, is underpinned by an embodied and affective logic in the production of meaning. Theorising cinematic sound, therefore, must entail how the auditory induction of affect produces embodied meaning.

5.4 Cinematic sound and the induction of affect My proposed embodied simulation account of cinematic sound uses Gallese’s ES theory to integrate three additional models to explain how cinematic sound elicits and structures affective content. The first is the BRECVEMA framework (Juslin 2013b; Juslin, Barradas & Eerola 2015; Juslin, Harmat & Eerola 2013; Juslin et al. 2010; Juslin & Västfjäll 2008) which models the underlying mechanisms for the auditory induction of emotion. The second is the mood-cue approach (Smith, GM 1999, 2003a) which models the multimodal induction of affect in the context of cinematic imagery. The third is the emotional communication model (Bartsch & Hübner 2005) which describes how neurobiology supports the social construction of emotion. Taken together, this composite “embodied simulation” approach provides a framework within which multiple models of affect induction and representation may be nested and scaffolded; it describes cinematic sound (and media generally) as fundamentally

138 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. affective and body-based; and it provides the foundation for intercorporeity, intersubjectivity, and empathy.

5.4.1 BRECVEMA framework Music has long been valued for its ability to arouse strong emotions in the listener. While this commonsense observation appears uncontroversial, philosophical and psychological accounts of musical emotion have been marked by paradox and disagreement. Contrasting Klaus Scherer’s assertion that “music does not induce basic emotions” (Scherer, K 2003) with Jaak Panksepp and Gunther Bernatzky’s equally forthright assertion that it is “remarkable that any medium could so readily evoke all the basic emotions” (Panksepp & Bernatzky 2002, p. 134), Patrik Juslin and Daniel Västfjäll observe that when it comes to music and emotion the “literature presents a confusing picture with conflicting views on almost every topic in the field” (Juslin & Västfjäll 2008, p. 559). Furthermore, disagreement abounds as to the efficacy of music to arouse genuine emotions because they do not appear to conform to the dominant concept within emotion theory of cognitive appraisal which claims that emotions arise when an event is appraised as having the ability to block or further one’s needs or goals. Peter Kivy has gone so far as to say that if music does arouse genuine emotions then a “special explanation” (1990, pp. 148-149) is required. Juslin and Västfjäll take the view that music does, in fact, induce genuine emotions on the basis that it is “not meaningful to make a distinction between ‘mediated’ and ‘unmediated’ musical emotions” (Juslin et al. 2010, p. 620).

In seeking to explain the many paradoxes in the field, Juslin and Västfjäll undertook research into the underlying psychological mechanisms for the musical induction of emotions. Initially developed by Juslin & Västfjäll (Juslin & Västfjäll 2008), subsequently refined in collaboration with Simon Liljeström and Lars-Olov Lundqvist (Juslin et al. 2010) and then developed further by Juslin (Juslin 2013b), the result of this research is the BRECVEMA framework which consists of “eight mini-theories of music-and-emotion, where each mechanism constitutes a causal explanation at a psychological level” (Juslin 2013a, p. 283, emphasis in original).

The BRECVEMA framework for the musical induction of emotion is comprised of eight mechanisms (Juslin 2013b, pp. 241-244): i) brain stem reflex, ii) rhythmic entrainment,

139 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. iii) evaluative conditioning, iv) contagion (emotional contagion), v) visual imagery, vi) episodic memory, vii) musical expectancy, and viii) aesthetic judgement. The acronym BRECVEMA comes about because the mechanisms are listed in order of evolutionary development (Juslin 2013b, p. 242), beginning with the phylogenetically ancient brain stem reflexes and progressing to the most recently evolved mechanisms of musical expectancy and aesthetic judgement. By way of a précis, below is a summary drawn from Juslin and colleagues (Juslin 2013b, pp. 241-244; Juslin et al. 2010, pp. 619-623):

Brain stem reflex (Juslin et al. 2010, pp. 620-621) denotes an emotion induction mechanism which autonomically parses the auditory environment for significant changes212. Acoustic features (such as sudden stimulus onset, loudness, dissonance, and fast temporal patterns) signal important or threatening events, and cue this reflex in the control of arousal, breathing, heart rate, attention, and movement. Brain stem reflexes are exceptionally rapid, automatic and innate. Shaped by ecological forces over an evolutionarily significant period of time, brain stem reflexes are the most phylogenetically ancient of the BRECVEMA mechanisms, and represent auditory perception at its most fundamental and robust, and, possibly, upon which other auditory specialisations later scaffolded. What is significant about brain stem reflexes for an embodied simulation account of cinematic sound is their capacity to cue the body for action without any input from consciousness.

Rhythmic entrainment (Juslin et al. 2010, p. 621) denotes an emotion induction mechanism which impels a listener’s internal body rhythm, such as breathing or heart rate, to synchronise to an external rhythm supplied by a musical pulse213. Moreover, the listener’s heart rate or respiration may be driven, within limits, by changes in that pulse, and because entrainment provides a “basis for timing in sensory-motor coordination and metre perception” it appears to coordinate the efficiency of physical work (Juslin et al. 2010, p. 621). This synchronisation process is progressive, and does not allow for rapid induction as is the case with brain stem reflex. In Chapter 7, rhythmic entrainment (Juslin et al. 2010, p. 621) is discussed in relation to the psychoacoustic phenomenon of ‘musical chills’ (Blood & Zatorre 2001; Blood et al. 1999; Gabrielsson 2001; Halpern, Blake & Hillenbrand 1986; Harrison & Loui 2014; Panksepp 1995; Sloboda 1991), ‘the drop’ in electronic dance music (Solberg 2014), and is connected to the concept of proto-narrative as an affective template.

140 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Evaluative conditioning (Juslin et al. 2010, p. 622) denotes an emotion induction mechanism which drives the pairing of a musical and non-musical stimulus no matter the valency. Evaluative conditioning can induce emotions outside of awareness, and is related to Zajonc’s mere exposure effect (Zajonc 2001). However, unlike classical conditioning, evaluative conditioning is resistant to extinction.

Emotional contagion (Juslin et al. 2010, p. 622) denotes an induction mechanism which drives an internal imitation of an emotional expression perceived within a musical source. Emotional contagion is known to occur not only in facial expressions (Hatfield, Cacioppo & Rapson 1993, 1994) but also speech (Neumann & Strack 2000), and the mechanism is proposed to activate in response to voice-like qualities contained within music. The mechanism produces internal mimicry of the perceived emotion, although no vocal sound need be expressed by the listener. Of significance to an embodied simulation account is the preliminary evidence which suggests this transfer of emotion occurs through activation of neural regions involved in “pre-motor representations for vocal sound production” implicating the involvement of the MNS (Juslin 2013b, pp. 241-242).

Visual imagery (Juslin et al. 2010, pp. 622-623) denotes a mechanism of emotion induction arising from the interaction of visual imagery and the music which spontaneously produces it. The exact nature of this mechanism is yet to be fully explicated but Juslin and colleagues, citing Lakoff and Johnson (1980a), suggest that listeners “conceptualize the musical structure through a metaphorical non-verbal mapping between the music and ‘image schemata’ grounded in bodily experience” (Juslin et al. 2010, p. 622). Furthermore, listeners appear to “respond to mental images in much the same way as they would to the corresponding stimuli in the ‘real’ world” (Juslin et al. 2010, p. 623), and so suggests an important mechanism for the creation and reception of cinematic imagery.

Episodic memory (Juslin et al. 2010, p. 623) denotes emotion induction via a musically triggered retrieval from memory of a significant event in the listener’s life. This mechanism is particularly vivid because the memory also retrieves the “physiological response patterns to the original events … along with the experiential contents” (Juslin et al. 2010, p. 623) suggesting embodied processes are at play. This mechanism is profoundly autobiographical, and appears to be central to the development of self-identity in adolescence.

141 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Musical expectancy denotes a mechanism whereby an emotion is induced because “a specific feature of the music violates, delays, or confirms the listener's expectations about the continuation of the music” (Juslin et al. 2010, p. 623). The knowledge of musical idioms and their attendant expectations develops over the course of an individual’s sociocultural listening history, and so will vary from listener to listener.

Aesthetic judgement (Juslin 2013b, p. 246) is the final mechanism to be included in the BRECVEMA framework. Whereas the original BRECVEM mechanisms have a long period of evolutionary development, aesthetic judgement is a recently evolved capacity. Aesthetic judgement also represents a significant difference in type from the original seven mechanisms. This difference turns on the distinction between ‘everyday emotions’ and ‘aesthetic emotions’ (Juslin 2013b, p. 237). Everyday emotions are what emerge from the first seven BRECVEM mechanisms but aesthetic emotions arise differently. An aesthetic judgment “is assumed to rely more on ‘higher’ cognitive functions, domain-relevant knowledge, and a fluid, individualized process that may change across time and context … aesthetic judgments are strongly (although not exclusively) influenced by cultural variables” (Juslin 2013b, p. 247). The mechanism of aesthetic judgement deals with a blend of autobiographical experience and socially learned and culturally transmitted behaviour. When compared to the original seven BRECVEM mechanisms, aesthetic judgement is highly volatile and variable, and could be (ungenerously) characterised as epiphenomenal.

The malleability of the mechanism implies it may be only loosely bound by the corporeal logic which shaped the original BRECVEM mechanisms. However, according to Robert Boyd and Peter Richerson “nothing about culture makes sense except in the light of evolution”214 (Richerson, Peter J. & Boyd 2005b). This is to say, culture is a biological strategy for survival. This insight places biology and culture within the same evolutionary framework, and is theorised variously as biocultural evolution (Cavalli-Sforza & Feldman 1981), gene-culture coevolution (Boyd & Richerson 1985, 2005; Laland, Kumm & Feldman 1995; Richerson, Peter J. & Boyd 2005a), and dual inheritance theory: the idea that genetic and cultural evolution interact to produce human behaviour215. This key insight – that biology and culture interact within the same evolutionary framework – is central to the embodied cognition approach to cinematic media I develop within this thesis.

Brain stem 1 Focusing Extreme or (Cued) reflex attention on rapidly changing sensorimotor potentially basic acoustic representations important characteristics changes or events in the close environment

Rhythmic 2 Facilitating Periodic pulses (Cued) entrainment motor in rhythms, sensorimotor coordination in especially representations physical work around 2Hz tasks

Evaluative 3 Being able to Co-variation (Cued) conditioning associate objects between events associative or events with representations positive and negative outcomes

Contagion 4 Enhancing group Emotional motor (Cued) cohesion and expression sensorimotor social reminiscent of representations interaction, e.g. human voices between mother and infant

Visual imagery 5 Permitting Self-conjured (Detached) internal visual images pictorial simulations of representations events that substitute for overt and risky actions

Episodic 6 Allowing Personal events (Detached) memory conscious in particular hierarchically recollections of places and at organized previous events particular times schematic and and binding the pictorial self to reality representations

Musical 7 Facilitating Syntactic (Detached) expectancy symbolic information schematic and language with a hierarchical complex representations semantics

Brain stem General arousal, Present High Low reflex surprise

Rhythmic General arousal, Present Low Low entrainment feelings of communion

Evaluative Basic emotions Present High Low conditioning

Contagion Basic emotions Present High Low

Visual imagery All possible Omnidirectional Low High emotions

Episodic All possible Past Low Medium memory emotions, but especially nostalgia

Musical Interest, anxiety, Present/Future Medium Low expectancy surprise, chills, hope, disappointment

Figure 5.1. The original seven BRECVEM mechanisms for the musical induction of emotion. Note that this model does not include the final mechanism of aesthetic judgement. Table adapted from Juslin (2013b, pp. 243-244). A more complete set of tables is supplied in Appendix B.

The BRECVEMA framework identifies specific mechanisms for the musical induction of emotion but it also offers insights for a theory of cinematic sound. Firstly, sensorimotor processes implicated in embodied simulation also appear to be involved in numerous BRECVEMA mechanisms, particularly brain stem reflexes, rhythmic entrainment, emotional contagion, visual imagery, and episodic memory. Secondly, a crossmodal mechanism is implicated in the sonic induction of emotional visual imagery. Thirdly, the eight BRECVEMA mechanisms yield a scalability from neurobiology to autobiography and social and cultural cognition. Lastly, the BRECVEMA framework provides insight into how cinematic sound, when mobilised through a bottom-up perceptual design strategy, activates a range of nonconscious mechanisms for the sonic induction of affect. The BRECVEMA framework acknowledges how “biological, socio-cultural, historical, and philosophical perspectives may all contribute important insights to the development of a psychological

144 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. theory of emotions and aesthetic judgments” (Juslin 2013a, p. 283), and these insights are used in Chapter 7 to examine the design of sound in a number of mediaworks.

The BRECVEMA framework presents itself as a strong candidate upon which to build a theory of cinematic sound. However, a slight complication arises in adopting the framework. Developed specifically in relation to the musical induction of affect, the BRECVEMA framework may not capture all possible sonic induction mechanisms216. As rich a world as music may produce, I assume music to be only a fraction of the possibilities of a sounding universe. However, whatever sonically exists outside the boundaries of music must exist within a broader evolutionary context217. Indeed, as Juslin and Västfjäll acknowledge, “music evokes emotions through mechanisms that are not unique to music” (Juslin & Västfjäll 2008, p. 559), and many of the BRECVEMA framework’s mechanisms “involve low-level processes that developed before music even existed, and that consequently don’t treat ‘music’ as a distinct type of event” (Juslin 2013b, p. 241). In all likelihood, the BRECVEMA framework may omit mechanisms which are primarily ecological in value, but it is with the general understanding of “music as sound in the most basic sense” (Juslin et al. 2010, p. 620, emphasis in original) that I import the BRECVEMA framework into my embodied simulation account of cinematic sound. With these complications in mind, and aware of the need for an ongoing development of the BRECVEMA framework, I will adapt and supplement the framework’s precepts when addressing specific issues around sound design for cinematic media.

Lastly, the BRECVEMA framework poses something of a challenge to the first wave of cognitive film theory as represented by David Bordwell, Noël Carroll, Carl Plantinga, and Ed Tan, among others. Cognitive film theory of this kind has tended to focus upon narrative cinema, and places great emphasis upon the explanatory power of cognitive appraisal theories of emotion and Theory of Mind. The BRECVEMA framework appears to stand outside this paradigm, supporting the notion that nonconscious affective mechanisms are mobilised by audition, and these mechanisms may be manipulated by cinematic sound design in the production of Feeling of Body and proto-narrative even before they become available to cognitive appraisal and narrative comprehension218. It is affective structure of this kind that is actualised in the mood-cue approach to cinema.

145 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 5.4.2 Mood-cue approach The mood-cue approach, developed by Greg M. Smith (Smith, GM 1999, 2003a), describes a general design strategy by which audiovisual imagery engages the audience’s affective system to induct mood and cue emotions. This is a fundamental cinematic function because, as Smith argues, the “primary emotive effect of film is to create mood” (Smith, GM 2003b, p. 42). Although devised for cinema, Smith notes the mood-cue approach may also be applicable to other audiovisual narrative media219 (Smith, GM 2003b, see note 5, p. 195).

A feature of Smith’s mood-cue approach is its description of cinematic media as a type of affective ecology: an experience of mood increases the likelihood of experiencing an emotion proper which further sustains (or alters) an experience of mood (Smith, GM 2003b, p. 42). Experiencing affect, as discussed in Chapter 4, performs a sense-making function because it forges a meaningful relationship between the audience and the objects, actions and characters within the cinematic environment.

Mood induction may be brought about via large-scale emotional cuing delivered redundantly through multiple channels of audiovisual design. Cinematography, visual effects, picture montage, production design, costume and make-up, music score, and sound design are just some of the audiovisual design channels through which affective content may be created and delivered (Smith, GM 2003b, p. 42). The strategy of redundancy in emotional cuing increases the possibility of effective communication with audience members who differ in their sensitivities and capacities: if one mode of cuing fails a different mode may be successful in the communication of an “appropriate emotional orientation” to the mediawork (Smith, GM 2003b, p. 43). Furthermore, the generation of mood through stylistic cuing is vital because narrative alone cannot sustain emotional experience. As Smith agues,

Few texts can rely only on narratively significant moments to provide mood- sustaining emotion. Most have to provide markers to shore up the mood the text has created, even if the markers have little or no effect on the overt diegetic aim: the character’s achievement of a goal. (Smith, GM 2003b, p. 45)

Smith usefully develops a number of terms when detailing how filmic mood-cue functions: terms such as associative network, input channel, emotion node, emotion cue, and emotion marker. The associative network model (Smith, GM 2003b, pp. 29-31) recognises the human

146 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. emotion system is comprised of multiple subsystems of which only the engagement of the phylogenetically ancient limbic system is essential for emotion to arise. In Smith’s associative network model

the various components of the emotion system are connected by a series of associative links. Emotions (the nodes of the system) are tied to particular thoughts and memories as well as patterns of physiological reactions. Conscious cognitions (memories, social mores, emotion labels, etc.), autonomic and central nervous system patterns, action tendencies, vocalizations, and facial patterns are all interrelated. (Smith, GM 2003b, p. 29)

An input channel (Smith, GM 2003b, p. 30) is a source of information to the emotion system, usually sensory. Emotions, on Smith’s account, are “nodes” (Smith, GM 2003b, p. 30) which interconnect the body’s emotional subsystems. The emotion system, because it accepts and processes multiple input channels in parallel, is capable of generating a vast network of associative connections. Thus,

[b]ecause emotions can be evoked using a wide range of stimuli linked in an associative network, films can use the full range of perceptual cues to evoke emotion … [including] facial expression, figure movement, dialogue, vocal expression and tone, costume, sound, music, lighting, mise-en-scène, set design, editing, camera (angle, distance, movement), depth of field, character qualities and histories, and narrative situation. (Smith, GM 2003b, p. 42)

An emotion cue (Smith, GM 2003b, pp. 29 and 42) is a packet of perceptual information which activates the emotion system.

Cues are the smallest unit for analyzing a text’s emotional appeals. Emotion cues are the building blocks that are used to create the larger structures such as emotion markers. Mood is sustained by a succession of cues, some of which are organized into larger structures (narrative obstacles, emotion markers), some of which are not. (Smith, GM 2003b, p. 47)

147 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Emotion markers are “configurations of highly visible textual cues for the primary purpose of eliciting brief moments of emotion” (Smith, GM 2003b, p. 44). The more densely cued the flow of audiovisual imagery, the more likely an experience of emotion is to occur. However, it should be noted that Smith makes a differentiation between what acts affectively and what acts narratively, and in so doing intimates the existence of two parallel (but interacting) structures.

The emotion marker is not there simply to advance or retard the narrative’s progress. Neither is the emotion marker an informative device offering more detail about the story or offering authorial commentary on the diegesis. The primary purpose of an emotion marker is to generate a brief burst of emotion. Often such moments could be excised from a film with little or no impact on the achievement of narrative goals or the state of story information. (Smith, GM 2003b, pp. 44-45)

Smith’s functional description of an emotion marker echoes Tom Gunning’s identification of a “sudden burst of presence” in the affective non-narrativity of cinema-of-attractions (Gunning 1993, p. 6). These observations by Smith and Gunning indicate the existence of a powerful interrelationship between sensation, emotion and presence. “Emotions”, reflect Rosa Baños and her colleagues, “may play a role both as determinants and consequences of presence” (Baños et al. 2004, p. 735), and this appears to be true for both real and virtual worlds. In experiments with virtual environments, Giuseppe Riva and associates found a “circular interaction between presence and emotions: on one side, the feeling of presence was greater in the “emotional” environments; on the other side, the emotional state was influenced by the level of presence” (Riva et al. 2007, p. 45). Furthermore, in discussing the interrelationship between emotion, presence and consciousness, Riva, Waterworth & Waterworth observe that in “evolutionary (and neurological) terms, sensation led to perception, and presence emerged as the feeling (in core consciousness) of being in an external, perceptible world in which things happen in relation to the organism” (Riva, Waterworth & Waterworth 2004, p. 414). This effect is robust for both real and virtual environments. Taken together, these findings suggest that cinematic sound design – considered as designed sensory information – creates a stream of emotion cues and markers which, in turn, create a feeling of presence which, in turn, generates consciousness.

148 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Smith’s mood-cue approach appears to acknowledge two impulses within contemporary emotion theory which alternately conceptualise human emotions as arising from bottom-up neurobiological and top-down cultural processing. I propose these dual impulses produce parallel (but interacting) structures within cinematic media, what I describe in Chapter 7 as perceptual design and narrative design. Referencing Stein & Trabasso (Stein, NL & Trabasso 1992), Smith claims “that because emotions are usually goal-oriented states, much cultural variation comes from cultures placing different values on particular goals” (Smith, GM 2003b, p. 35).

Part of the socializing work of a culture is to provide its members with scripts for expressing and experiencing emotion, prescribing what sorts of circumstances elicit emotions, who should feel, and the boundaries of appropriate feeling and expression. (Smith, GM 2003b, pp. 34-35)

Although Smith acknowledges the power of culturally determined emotions, the mood-cue approach emphasises the so-called ‘hardwired’ nature of neurobiological processing as the genesis of affect. Some emotions, as a consequence of our shared biology, “transcend culture” (Smith, GM 2003b, p. 35), and the issue turns on ascertaining which is which within the biocultural matrix. However, Smith observes that by grounding “socially constructed prototypes in certain simple materialities, societies are able to justify their prescriptions on emotion as if they were ‘natural’” (Smith, GM 2003b, p. 35). In recognising the interplay between ‘universal’ and ‘cultural’ emotions, Smith acknowledges the “sociocultural network of emotional associations that individuals have constructed has more bearing on most everyday emotional experiences than any ‘universals’” (Smith, GM 2003b, p. 35). Smith’s comment speaks to the human level at which the sociocultural construction of emotion registers; however, “[s]ociocultural influences shape our emotions, but they cannot fundamentally remake the construction of the emotion system” (Smith, GM 2003b, p. 35). In short, there are constraints and determinations placed upon culture by biology.

Smith (Smith, GM 2003b, pp. 65-81) claims his mood-cue approach stands in contrast to other cognitivist accounts of filmic emotion such as the cognitive philosophical stance taken by Noël Carroll (1988b) and the cognitive psychological stance of Ed Tan (1996) and Torben Grodal (1997). Although Smith’s mood-cue approach is under-appreciated220 within the literature, it should not be glossed over; it offers powerful differences to the approaches taken

149 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. by Carroll, Tan, and Grodal. The cognitivist accounts supplied by Carroll and Tan221 provide what amounts to social simulation theories of cinema because they strive to explain a Theory of Mind for (fictional) characters, but Smith’s contrasting account supplies a perceptual simulation theory because he focuses upon bottom-up processes that will, ultimately, produce a ‘Feeling of Body’. The differential between these two types of cognitive film theory, which amounts to the difference between that which privileges Theory of Mind (ToM) versus that which privileges Feeling of Body (FoB), motivates the development of my conception of narrative immersion, perceptual immersion, and proto-narrative, and which I illustrate with specific examples in Chapter 7.

Whereas the major cognitive film theories focus upon the role of emotion (as cognitive appraisal) within narrative, the mood-cue approach is equally applicable to both narrative and non-narrative cinema because it explains the function of media emotion not in terms of character and diegesis but in terms of audiovisual design. “An important advantage”, Smith claims, “of the mood-cue approach over [Noël] Carroll’s is that it can provide explanations of filmic emotion without relying solely on character-oriented cues” (Smith, GM 2003b, p. 70). Indeed, the mood-cue approach suggests how cinema-of-attractions and post-classical cinema, with their foregrounding of audiovisual event and style, may be theorised because it explains how perceptual design engages the human affective system without recourse to narrative. This distinction is sharply drawn when Smith observes:

Narrative provides a series of diegetic goals and obstacles, and goal achievements and obstacles frequently provide the necessary mood-reinforcing pay-offs. We rejoice when the protagonist achieves a goal or subgoal; we are sad, fearful, or anxious when a goal is frustrated. Goals and obstacles are highly foregrounded in the narrative, and so they create highly marked opportunities for moments that are significant both narratively and emotionally.

Almost every narratively significant moment has the potential to provide some emotional payoff, but not every emotional payoff is narratively significant. (Smith, GM 2003b, p. 44)

In conclusion, the mood-cue approach is relevant to an embodied simulation account of cinematic media for several reasons. Firstly, it describes the direct role of cinematic imagery

150 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. in the induction of core affect and mood222. The mood-cue approach can be understood to recruit and scaffold upon the auditory induction mechanisms described in the BRECVEMA framework, particularly the mechanisms of brain stem reflex, rhythmic entrainment, evaluative conditioning, emotional contagion, visual imagery, and episodic memory. Secondly, it contributes additional nuance and sophistication to these mechanisms by adding the dimensions of visually induced affect and crossmodal interaction223. Thirdly, like the BRECVEMA framework, the mood-cue approach describes how nonconscious affect may be mobilised and organised through a bottom-up design strategy which I have labelled perceptual design. What emerges from the coordination of emotion cues into emotion markers is affective sequencing and flow. This flow registers as mood, but, more than this, it coalesces as proto-narrative.

Although Smith does not reference James A. Russell’s concept, core affect appears to operationalise filmic mood-cue. Core affect, as described in Chapter 4, is a pre-conceptual process which may be “experienced as free-floating (mood) or can be attributed to some cause” (Russell 2003, p. 145). This is to say, once generated, core affect may be guided so as to be causally (mis)attributed to target characters, objects, locations and scenarios within the cinematic environment and thus absorbed into the diegesis, or simply remain free-floating as mood. The mood-cue approach theorises how an affective experience may arise solely from cinematic imagery and indicates how perceptual design functions in parallel with – and often supports – narrative structure and comprehension.

At first glance, both the BRECVEMA framework and the mood-cue approach appear detached from an everyday movie experience of story. What is required to harmonise these decidedly subpersonal accounts with human-scale understanding is a framework which integrates bottom-up with top-down affective processes. Emotional communication theory supplies such an integrative framework.

5.4.3 Emotional communication theory Emotional communication theory, developed by Anne Bartsch and Susanne Hübner (2004, 2005; 2005) and described in Chapter 4, proposes human communication to be not only concerned with exchanging information but also with exchanging emotions as a process of “mutual influence” between communicating partners (Bartsch 2004, p. 1). In observing that emotion theories are traditionally grouped into four competing accounts – e.g., neuroscience

151 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. models, appraisal theories, prototype approaches, and social constructionist theories. Bartsch & Hübner reconcile these accounts by understanding them as descriptions of emotion at different levels of cognitive complexity. The theory describes a model which contains three vertically integrated but interacting levels where “[l]evel one consists of the reciprocal activation of emotional brain systems, level two consists of the reciprocal activation of emotional scripts, and level three consists of the symbolic negotiation of emotions” (Bartsch & Hübner 2005, p. 4). A hallmark of the theory is its capacity to integrate the neurobiological, personal, social and cultural levels of emotional experience.

Central to Bartsch & Hübner’s theory is the recognition that top-down mechanisms of emotion are rooted in and reliant upon bottom-up affective processing, claiming that “superordinate levels of emotional communication involve automatically communication on the more basic levels”, but “emotional communication on a symbolic level will not work if it does not activate nonverbal emotion scripts, which, in turn, depends on the activation of emotional brain systems” (Bartsch & Hübner 2005, pp. 4-5). Symbolic communication of emotion at the level of social construction cannot function without the body’s affective neurobiology to support it, whereas the reverse is not true. These observations echo Mark Johnson’s theory of embodied meaning (Johnson, M 1997, 2007b) which argues that even the highest levels of complexity found in abstract thinking have their basis in the lowest levels of the biological. It is also significant to note how Bartsch and Hübner invoke in the phrase “reciprocal activation of emotional brain systems” the property of resonance which is at the core of Gallese’s theory of embodied simulation.

Emotional communication theory provides a framework by which to explain how moral emotions and reasoning might emerge from the basic affective structures created by cinematic imagery. Bartsch’s work on embodied metaphor and the function of meta-emotions224 within art and entertainment (Bartsch 2008, 2010; Bartsch & Oliver 2011) usefully connects cognitive neuroscience and social constructionism, two disciplinary camps which have traditionally found it difficult to engage.

In conclusion, the significance of Bartsch & Hübner’s emotional communication theory to the proposed embodied simulation account of cinematic sound is its ability to integrate the profoundly subpersonal nature of affective processing detailed by the BRECVEMA framework

152 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. and mood-cue approach with the more traditional concerns of the humanities which tend to focus upon the human scale of social and cultural practices.

5.5 Summary: embodied simulation and cinematic media In this chapter an embodied simulation account of cinematic sound has been set out. It assumes an analogue between real and virtual worlds, and places embodied affect at the centre of the meaning-making process both in the creation and reception of cinematic media. The account is comprised of four components. Firstly, it uses Vittorio Gallese’s theory of embodied simulation (Gallese 2005) as a unifying macro framework. The explanatory power of this framework lies in binding sensation, emotion, cognition, consciousness, intersubjectivity and empathy through embodied processes and formats (Goldman & de Vignemont 2009). Affect is the lingua franca between these processes, and is central to embodied cognition. Nested within this macro framework are three models to explain the induction and structuring of affect arising from mediated imagery: i) the BRECVEMA framework developed by Patrik Juslin, Daniel Västfjäll and colleagues (Juslin 2013b; Juslin et al. 2010; Juslin & Västfjäll 2008); ii) the mood-cue approach developed by Greg M. Smith (Smith, GM 1999, 2003a); and iii) emotional communication theory developed by Anne Bartsch and Susanne Hübner (Bartsch 2004, 2005; Bartsch & Hübner 2005).

The BRECVEMA framework describes eight mechanisms by which affect may be inducted by sound, and scales from the neurobiological to the social. These mechanisms range from the evolutionarily ancient to the more recently developed, from the innate (brain stem reflexes) to the enculturated (aesthetic responses). The mood-cue approach models the affective nature and function of cinematic imagery. It details how the look and sound of cinematic imagery generates mood, feeling and emotion and how they interact to sustain an affective ecology. According to the approach, affective cuing may proceed without reference to character identification or narrative goals, and therefore indicates the existence of a strongly affective bottom-up cinematic process. Emotional communication theory describes the scaffolding biocultural nature of emotional experience, and operationalises this within cinematic media.

Standard theories within Film Studies place great emphasis upon visual imagery and narrative to the detriment of the aural and non-narrative dimensions of the cinematic experience. My proposed embodied simulation account shows how cinematic sound generates affect and

153 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. creates embodied meaning in its own right as well as in interaction with visual imagery. Audition is at the heart of meaning-making because the MNS integrates auditory, visual and motor systems, and so is instrumental in the processes through which higher cognitive functions such as narrativity, rationality and abstract thought are constructed. However, my proposed ES account has thus far dwelled upon nonconscious affect and non-narrative. To avoid repeating in reverse standard film theory’s sin of elision, my proposed framework must be capable incorporating the human levels of consciousness and narrative. I will address these issues, with reference to specific theories of consciousness, in the next chapter.

6.1 Consciousness, self, and narrative Before turning to apply my proposed framework to actual mediaworks I feel the need to provide lines of connection between embodied cognition and notions of narrative and story. One reason for doing this is to address a possible criticism my approach might attract from the direction of standard film theory which tends to be narrative-centric. In the account I am presenting, as should be abundantly clear, narrative is not central. Even so, my account must offer lines of inquiry to explain the experience of cinematic narrative, and it does this via neurobiological theories of consciousness. Although this chapter does not immediately appear concerned with developing an ES account of cinematic sound, it explains how the induction of affect provides an underlying structure upon which consciousness can scaffold, and in doing so reveals linkages to narrative and story. The account extends from the production of affect and embodied meaning derived from sensory information up to the point of transduction of this affect in the generation of story. It fleshes out the proposed cinematic system of perceptual and narrative immersion by building upon the embodied cognition framework developed in Chapters 4 and 5 to show how sound-induced affect affords the production of consciousness and hence narrative. Theories of embodied cognition present an account of ourselves so peculiar and unfamiliar they appear almost impossible to reconcile with our everyday experience. We are, as Timothy Wilson says, “strangers to ourselves” (Wilson, TD 2002). We glimpse this strangeness when we confront the notion that approximately 95% of all thought is nonconscious, and this activity “below the surface of conscious awareness shapes and structures all conscious thought” (Lakoff & Johnson 1999, p. 13). How can the automaticity of these vast assemblages of sub-personal processes be squared with our everday feeling of a unified conscious self? Fauconnier and Turner call this a problem of “achieving human scale” (Fauconnier & Turner 2002, p. 312, 319 and 346), or what Slingerland calls “human-level structures of meaning” (Slingerland 2008b, 2009). How may an embodied simulation account of cinematic media be reconciled with the everyday human experience of being a narrative self?

In this chapter, the neurobiological framework for emotion and cognition developed in Chapters 4 and 5 is brought to bear upon the familiar concerns of standard film theory. It

155 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. relates embodied cognition to consciousness and, by extension, to narrative, the self, and culture. It explores the types of consciousness human beings appear to have, the special function of affect and narrative in constructing the self and conscious thought, and proposes an interrelationship between perceptual consciousness, presence, cinematic spectacle and narrative.

6.2 Explicating consciousness David Chalmers has famously categorised questions of consciousness into “easy” and “hard problems” (Chalmers 1995, p. 200). To grapple with the easy problems one must explain the mechanisms by which we differentiate, categorise and react to the environment, integrate information, access and report our mental and internal states, focus our attention, and control our behaviour whereas to confront the hard problems is to explain how it is we have corporeal experience in the way that we do (or at all) (Chalmers 1995, p. 200). Very clearly, I am confronting the hard problem of consciousness when theorising the aesthetic and meaning- making capacities of cinematic sound. The hard problem is undoubtedly complex and historically intractable. It is the site of the putative mind–body split enshrined in Cartesian dualism. The pervasive nature of the hard problem so dominates the philosophical literature on consciousness it has given Jesse Prinz cause to lament it as “an excuse to work and re- work familiar positions on the mind–body problem” (Prinz, J 2005, p. 381). How may we begin to close this infamous “explanatory gap”?225

6.2.1 Approaches to theorising consciousness Consciousness Studies is a serpentine interdisciplinary field, and this thesis affords only limited opportunity to survey key concepts and issues when relating narrative to an embodied simulation account of cinematic media. To constrain the conceptual landscape I shall follow philosopher Ned Block (Block 2009) in briefly reviewing three major competing theoretical approaches which dominate the field – Higher-Order Thought theory, Global Workspace theory, and Biological Framework theory – and why the biological approach is the most significant.

Higher-Order Thought theory (HOT), championed by David Rosenthal (Rosenthal 1986, 1990) and William Lycan (Lycan 1996, 2001) proposes phenomenal experience becomes conscious only when monitored by another state. It is characterised as supremely top-down.

156 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. HOT theories, according to Block, suffer an inherent weakness in confronting the explanatory gap because they are “geared to the cognitive and representational aspect of consciousness” (Block 2009, p. 1114) but fail when confronted with explaining the phenomenal states themselves.

Global Workspace theory (GWT) developed by Bernard Baars (Baars 1988, 1997) and extended by Stanislas Dehaene and colleagues (Dehaene & Changeux 2004; Dehaene et al. 2006; Dehaene & Naccache 2001) is the second major approach to consciousness. GWT proposes that

at any given time, many modular cerebral networks are active in parallel and process information in an unconscious manner. An information becomes conscious, however, if the neural population that represents it is mobilized by top-down attentional amplification into a brain-scale state of coherent activity … [G]lobal availability of information through the workspace is what we subjectively experience as a conscious state. (Dehaene & Naccache 2001, p. 1)

Block observes that GWT provides a compelling description of how conscious thought consumes perceptual representations in the production (and subsequent self-consumption) of higher order representations. However, the weakness of GWT is that neurobiology is not inherently demanded because

the substantive claims of the model abstract away from neuronal details. Nothing in the model requires the electrochemical nature of actual neural signals. The architectural aspects of the model can just as easily be realized in silicon-based computers as in protoplasm. In this respect, the global workspace theory of consciousness is … characterized by an abstract structure that does not include the messy details of neuroscience. (Block 2009, p. 1111, italics in original)

The third major approach is Biological Framework theory which simply holds “consciousness is some sort of biological state of the brain” (Block 2009, p. 1112)226. In assessing the fitness of a biological account, Block observes:

157 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. [G]lobal broadcasting and higher order thought are what consciousness does rather than what consciousness is. That is, one function of consciousness on the biological view is to promote global broadcasting, and global broadcasting in some but not all cases can lead to higher order thought. Further, according to the biological view, both the global workspace and higher-order thought views leave out too many details of the actual working of the brain … (Block 2009, p. 1113, emphasis in original)

As should be apparent, the competitors to the biological account are profoundly nonbiological, having more of their inspiration in the computer model of the mind of the 1960s and 1970s … (Block 2009, p. 1119)

Taken together, these assessments lead Block to maintain “it is hard to avoid the impression that the biology of the brain is what matters to consciousness – at least the kind we have” (Block 2009, p. 1120). Clearly, then, Block’s argument harmonises with theories of embodied cognition which similarly demand a neurobiological account of consciousness227.

A serious obstacle to the study of consciousness is that it is a “mongrel concept” which “connotes a number of different concepts and denotes a number of different phenomena” (Block 1995, p. 227). There are many types of consciousness. For the sake of brevity, I shall focus upon the most commonly cited typologies – Ned Block’s phenomenal and access consciousness, António Damásio’s core and extended consciousness, and Marc Jeannerod’s embodied and narrative self – in drawing out the relationship between cinematic imagery, narrative and consciousness. My reason for doing this, aside from containing a fractious debate to the most heavily cited forms of consciousness, is to reveal their similarities and significance to what I describe in Chapter 7 as perceptual and narrative immersion.

6.2.2 Types of consciousness – Phenomenal and access consciousness (Ned Block) Block (1995, pp. 230-236) teases apart four discrete types of consciousness228 which he believes have been confounded: i) phenomenal-consciousness (P-consciousness), ii) access- consciousness (A-consciousness), as well as iii) self-consciousness (S-consciousness), and iv) monitoring-consciousness (M-consciousness). For the purposes of argument, I shall concentrate upon Block’s distinction between P-consciousness and A-consciousness. Bluntly put,

158 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. [p]henomenal consciousness is experience; the phenomenally conscious aspect of a state is what it is like to be in that state. The mark of access-consciousness, by contrast, is availability for use in reasoning and rationally guiding speech and action. (Block 1995, p. 227)

P-consciousness is somewhat elusive; sensed in experiential properties such as sensations, feelings and perceptions 229. Block assumes P-consciousness properties to be distinct from any cognitive, intentional, or functional property whereas A-consciousness is tied to the ‘accessible’ experience of higher cognitive function such as sequential thought, reasoning and reporting (Block 1995, p. 230). P-consciousness eludes succinct definition precisely because it offers only limited pregnability to A-consciousness, and A-consciousness is responsible for reporting. In contrast, a definition of A-consciousness is highly detailed precisely because this is where reasoning and reporting occur. Broadly speaking, the content of P-consciousness is the feeling of “what it is like” whereas A-consciousness is the cognitive manipulation of that content (Block 2009, p. 1113). Although P- and A-consciousness interact, it is difficult to determine the exact nature of this interaction because phenomenal experience is cognitively inaccessible and resides at the root of the explanatory gap (Levine 1983; Nagel, T 1974). Nevertheless, Block proposes one consequence of the interaction between P- and A- consciousness is a phenomenon he terms cognitive overflow (Block 2007b, 2007a, 2011b).

Cognitive overflow proposes “the capacity of phenomenal consciousness exceeds that of cognitive access” (Block 2011b, p. 567), and thus is not wholly available to conscious thought230. Cognitive overflow is related to issues of sensory information, bandwidth and compression. It is estimated that approximately 11 million bits of information flow into the buffers of human sensory memory every second (Dijksterhuis & Nordgren 2006, p. 97). Sensory memory is a high-capacity but extremely short-duration (50–500 milliseconds) form of memory. These buffers charge and discharge rapidly but only a fraction of the immense tranche of information stored in sensory memory is capable of being processed by conscious thought, and only then in a compressed format. The throughput of conscious thought is pitifully low and processing speed is dependant upon the type of task undertaken. Reading silently to oneself, for example, occurs at a processing speed of 45 bits/second while reading aloud is significantly lower at 30 bits/second, and during numerical calculation the rate plunges to a mere 12 bits/second (Dijksterhuis, Aarts & Smith 2005, p. 82). A severe bottleneck exists in the order of 220,000:1 between the bandwidths of phenomenal

159 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. consciousness and conscious thought. This is what Benjamin Kuipers calls “drinking from the firehose of experience” (Kuipers 2008). Conscious thought is vastly outstripped by the fast and wide nature of phenomenal consciousness (Kuipers 2008, p. 168) and so, for the serialising logic of conscious thought to operate efficiently, sensory information must be radically compressed. Gilles Fauconnier and Mark Turner propose compression of this kind gives rise to representation, image schema, metonymy, and metaphor (Fauconnier 2005; Turner 2006b, 2006a).

In the context of cinematic imagery, cognitive overflow provides a useful concept by which to gain a new perspective on Kristin Thompson’s notion of cinematic excess (Thompson, K 1977). If I am correct in extrapolating from Block, P-consciousness contains nonconscious perceptual information which is in excess of the capacity of narrative consciousness to render it. Similarly, cinematic excess is in excess of the requirements of story. When cinematic excess is considered from the perspective of nonconscious processes we are returned to a consideration of Zajonc’s thesis of affective primacy (Murphy & Zajonc 1993; Zajonc 1980, 1984) (discussed in Chapter 4) which argues core affect and unconscious emotion are dominant in meaning-making and the shaping and guiding of our perception of persons, objects and environments231. Counterintuitively, this inverts the orthodoxy in Film Studies that story is deep and spectacle is shallow, and leads to a replacement of the dualistic spectacle/narrative framing with an understanding that narrative is emergent from the same embodied processes from which spectacle is derived.

Furthermore, cognitive overflow, considered as processes which function before and below conscious awareness, also resonates with Tom Gunning’s concept of cinema-of-attractions. Cinematic attractions possess their own unique temporal structure and function in parallel to narrative yet are never extinguished by it (Gunning 1993, p. 4). Taken together, a special relationship suggests itself between the mechanisms of P-consciousness and cinema-of- attractions, cinematic excess and spectacle, and thus the design of cinematic sound. In Chapter 7 the concepts of cognitive overflow and cinematic excess are expanded through a stylistic analysis of In the Cut (Campion 2003).

6.2.3 Types of consciousness – Core and extended consciousness (António Damásio) The neurobiological theory of consciousness developed by António Damásio (Damásio, AR 1989b, 1994, 1998, 1999, 2003, 2010; Damásio, AR & Meyer 2009) aligns in many respects

160 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. with that proposed by Block. However, Damásio presents a much fleshier account of how consciousness arises. It is not possible to do justice to the depth of Damásio’s work in the space available but I do wish to draw attention to several insights concerning core and extended consciousness and their relevance to a theory of cinematic media. In particular is the central function of affect in producing the imagery of core consciousness, and the pulsed nature of that imagery. Within Damásio’s account, the brainstem provides the affective foundation upon which the mind and modern consciousness arises. The brainstem, as discussed in Chapter 5, is the first evolutionary mechanism of the BRECVEMA framework for the auditory induction of affect, and so Damásio’s account of consciousness offers a blueprint for the deep structure of sound design and cinematic media.

Biological value of consciousness Damásio’s account of consciousness (Damásio, AR 1999) is predicated upon a deceptively simple premise: the value of consciousness lay in its assistance to the maintenance of biological homeostasis. Homeostasis refers to the

coordinated and largely automated physiological reactions required to maintain steady internal states in a living organism. Homeostasis describes the automatic regulation of temperature, oxygen concentration, or pH in your body. … Curiously enough, emotions are part and parcel of the regulation … (Damásio, AR 1999, pp. 39-40)

The value of consciousness, on Damásio’s account, is its novel means for achieving homeostasis. Not that it replaces the evolutionary older mechanisms, such as those provided by the brainstem and hypothalamus. Consciousness operates in adjunct to these ancient systems by affording a new means to solve “different kinds of problems that are connected, nonetheless, to the problems solved by previously existing means of homeostatic regulation” (Damásio, AR 1999, p. 303).

Emotion and the scaffolding of consciousness A central feature of Damásio’s neurobiological account of consciousness is the role of affect: emotion and consciousness are inseparable because consciousness arises from the corporeal experience of emotion (Damásio, AR 1999, p. 16). On Damásio’s account, emotion denotes neural responses to qualia which are not available to consciousness (and so deviates from the definition of the term established in Chapter 4, resembling something more akin to the notion

161 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. of core affect). Emotion is taken by the organism as a “neural object” which acts upon the proto-self to generate core consciousness (Damásio, AR 1999, p. 79). Damásio considers this to be the originating ‘self’ mechanism which allows the mind to become conscious to itself.

Damásio dissociates human consciousness into a minimum of three distinct layers: i) the proto-self, ii) core consciousness, and iii) extended consciousness (Damásio, AR 1999, pp. 16 and 22). (Damásio also refers to core consciousness as the core self and extended consciousness as the autobiographical self (Damásio, AR 1999, p. 17).) The proto-self, core consciousness, and extended consciousness are organised as a scaffolding hierarchy.

The self is built in distinct steps grounded on the protoself. The first step is the generation of primordial feelings, the elementary feelings of existence that spring spontaneously from the protoself. Next is the core self. The core self is about action – specifically, about a relationship between the organism and the object. The core self unfolds in a sequence of images that describe an object engaging the protoself and modifying that protoself, including its primordial feelings. Finally, there is the autobiographical self. This self is defined in terms of biographical knowledge pertaining to the past as well as the anticipated future. … The protoself with its primordial feelings, and the core self, constitute a ‘material me.’ The autobiographical self, whose higher reaches embrace all aspects of one’s social persona, constitute a ‘social me’ and a ‘spiritual me.’ (Damásio, AR 2010, p. 24, emphasis in original)

Like Block’s concept of P- and A-consciousness, Damásio’s typology parses a ‘phenomenal’ and a ‘social’ self. Note that the proto-self produces affect of a very basic type, and this affect is consumed by core consciousness in the construction of its constituent images. What core consciousness actually images is the relationship between the body and the environment. Temporally, core consciousness is online because it is concerned with the here-and-now while extended consciousness is offline because it is concerned with consuming the images of core consciousness for the purpose of remembering the past and predicting the future. Core consciousness is the here-and-now which is drawn upon by extended consciousness to build the autobiographical self. Damásio’s model of scaffolding layers of self is reminiscent of Bartsch & Hübner’s scaffolded model of emotion (see Chapter 4). Although these accounts describe a vertical hierarchy which appears to privilege the more cognitively complex layer,

162 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. this is not the case; the higher levels cannot exist without the lower levels and are parasitic upon them232.

Proto-self The primitive nonconsciousness of the proto-self is, on Damásio’s account, “a coherent collection of neural patterns which map, moment by moment, the state of the physical structure of the organism” (Damásio, AR 1999, p. 153). It is directly responsible for the regulation of homeostasis and the maintenance of life. It “has no powers of perception and holds no knowledge”, and should not be confused with the conventional notion of the self (Damásio, AR 1999, p. 154). It is important to note that the proto-self “is not an interpreter; it is a reference” (Parvizi & Damásio 2001, p. 138), and upon this reference are scaffolded core and extended consciousness.

Core consciousness Core consciousness is produced as the proto-self interacts with the environment. It is a momentary but continually generated and pulsed sense of self that is “not only about the representation of objects and events, but also about the representation of the organism it belongs to, as the latter interacts with objects and events” (Damásio, AR & Meyer 2009, p. 5). Core consciousness materialises “when you confront an object … and discover automatically that the now-salient image of the object is formed in your perspective, belongs to you, and that you can even act on it” (Damásio, AR 1999, p. 126) thus generating a unified subjectivity that possesses a unique perspective of the objects it interacts with and adds to the autobiographical self (Damásio, AR & Meyer 2009, pp. 6-7). In this way, Damásio’s concept of core consciousness arising from organism–environment interaction parallels the claim by Varela, Thompson & Rosch (Varela, Thompson & Rosch 1991) that cognition emerges through enaction.

Core consciousness is profoundly rooted in the present. There is “no elsewhere, there is no before, and there is no after” (Damásio, AR 1999, p. 16). Delimited by an exceptionally narrow temporal window, core consciousness is produced in pulses, and the “interval between units is so small and the amount of parallel pulses so abundant, that we only register a continuous, whirring blur” (Damásio, AR 1999, p. 346). Thus, each pulse of core self is a momentary burst of perceptual presence that manifests as a flow of images233.

163 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Extended consciousness If core consciousness is the here-and-now, then extended consciousness “places that self in individual historical time, in a perspective of both the lived past and the anticipated future” (Damásio, AR & Meyer 2009, p. 6). In differentiating extended from core consciousness, Damásio notes

core consciousness is a simple, biological phenomenon; it has one single level of organization; it is stable across the lifetime of the organism; it is not exclusively human; and it is not dependent on conventional memory, working memory, reasoning, or language. On the other hand, extended consciousness is a complex biological phenomenon; it has several levels of organization; and it evolves across the lifetime of the organism. … It depends on conventional memory and working memory. When it attains its human peak, it is also enhanced by language. (Damásio, AR 1999, p. 16).

At the human level, the autobiographical self of extended consciousness is a “relatively stable collection of the unique facts that characterize a person” and which “depends on memories of past situations” (Damásio, AR & Meyer 2009, p. 6). It is important to note, however, that extended consciousness emerges from, and is reliant upon, core consciousness (Damásio, AR & Meyer 2009, pp. 6-7).

6.2.4 Types of consciousness – Embodied and narrative self (Marc Jeannerod) The model of consciousness developed by Marc Jeannerod parses the mind in ways similar to both Block and Damásio. Jeannerod, a neurophysiologist and experimental psychologist, foregrounds two general forms of consciousness which he labels the embodied self and the narrative self. The embodied self is a nonconscious but direct mode of action control while the narrative self performs an autobiographical function at a human level of social understanding. The embodied self, therefore, aligns with Block’s P-consciousness and Damásio’s core consciousness while the narrative self aligns with Block’s A-consciousness and Damásio’s extended consciousness.

Where Damásio’s account casts core consciousness as images generated in nonconsciousness too profligate for complete display in consciousness (Damásio, AR 2010, p. 185), Jeannerod is concerned with the temporal interplay between embodied and narrative consciousness. Crucial in Jeannerod’s account is the timing with which narrative consciousness arises which

164 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. suggests the narrative self comes too late to be causal of action and behavior. This insight is as counterintuitive as it is provocative, and rather than paraphrase, I quote Jeannerod’s own summary234:

The type of consciousness (in fact, of non-consciousness) that is linked to the experience of the embodied self … is discontinuous. It operates on a moment-to- moment basis … In contrast, the conscious sense of will that we may experience when we execute an action, which is at the origin of our narrative continuity, arises from the belief that our thoughts can have a causal influence on our behavior. … While we tend to perceive ourselves as causal, we actually ignore the cause from which our actions originate. … Because the conscious thought and the observed action are consistently associated, even though they may not be causally related, the narrative self tends to build a cause-and-effect story. The embodied self, in contrast, by avoiding conscious introspection, reaches simpler (and perhaps more secure) conclusions about an action … The role of consciousness should rather be to ensure the continuity of subjective experience across actions which are – by necessity – executed automatically. Because it reads behavior rather than starting it, consciousness represents a background mechanism for the cognitive rearrangement after the action is completed … [because we] tend to “tell more than we know,” this mechanism could have the role of establishing a declarative cognitive content about one’s own preferences, beliefs, or desires. (Jeannerod 2006, pp. 36-37)

The embodied self manifests situationally, and is opaque to conscious awareness235. It provides automatic and direct control of action and behaviour236. In contrast to the pulsed nature of the embodied self, the narrative self – like Damásio’s autobiographical self – possesses extended temporal duration. This extended duration is the basis for our continuity of subjective experience. Narrative consciousness, however, is a “background mechanism for the cognitive rearrangement after the action is completed”. It smooths into continuity the blizzard of nonconscious images and the pulse of presence emanating from the embodied self, and modifies and justifies these events to create a coherent narrative through the reverse- engineering of putative causes.

In short, the narrative self is experienced as human-level subjectivity but the agency it suggests is an illusion237. Consciousness takes time to emerge and therefore cannot be causal

165 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. of behaviour (Cleeremans & Sarrazin 2007, p. 181). The story-building of the narrative self occurs downstream, safely sequestered from online behaviour. As with Damásio’s extended consciousness, the narrative self is concerned with remembering the past and predicting the future. Its function is to serialise, contextualise and narrativise objects, persons and events to explain why things happened in the way they did and anticipate how they might happen in the future. This is the novel and unique kind of information narrative consciousness provides the human animal. Narrative knowledge may not be strictly true but, from an evolutionary perspective, what it yields is true enough to be valuable and adaptive.

Embodied and narrative forms of self offer inherent benefits and drawbacks. For example, a positive consequence of the embodied self is the actual production of phenomenal presence, but a negative side-effect is the brevity and discontinuity of these phenomenal chunks. In contrast, a positive consequence of the narrative self is the capacity to rearrange these phenomenal chunks into a smooth and (apparently) sequential flow of autobiography, but a negative side-effect is an inability to directly access the lived world. The narrative self is always and already sealed off from the present.

The embodied self processes vast amounts of sensory data to produce pulses of imagery which, in turn, produce presence. The narrative self, on the other hand, is sealed off from direct experience of the world yet is parasitic upon the embodied self because it merely “reads” behaviour. Narrative is a post-hoc fabulation which does not provide accurate perceptual-causal explanation. But what the narrative self offers is predictive power. It abstracts rules and builds scenario models from past experience, even if that past is only exceedingly recent. From these models it predicts possible outcomes when certain conditions arise in the yet-to-be-experienced future. The adaptive value of the narrative self and its serialising logic is its ability to manipulate, re-organise, and re-experience (even counterfactually).

Jeannerod’s claim for an interacting embodied and narrative self provides a theoretical basis for what I observe in cinematic media as the design strategies of perceptual and narrative immersion. (This schema is developed in detail in Chapter 7.) Jeannerod’s research also illustrates how the limits of consciousness are also the limits of narrative. The narrative self builds a story of cause-and-effect although this causality is illusory and often inaccurate (Jeannerod 2006, p. 37). This insight alone has far-reaching implications for Film Studies

166 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. because it undermines the authority of narrativity as the sine qua non of meaning. At minimum, it questions the conditions under which narrativity provides explanatory value, and suggests the need to re-evaluate the theoretical relationship between cinematic imagery and narrative.

6.3 Narrative consciousness as animal–culture interface The above accounts undermine the veracity of our everyday belief of narrative consciousness as a command-and-control centre to sense the environment, judge situations, formulate plans, and volitionally guide the body to achieve goals. This is the classical sandwich model of cognition238. Such a belief appears self-evident, and Western society’s folk-psychology is predicated upon such a view. And yet, as the research above has shown, it is a view that is almost certainly incorrect.

What is the purpose of consciousness if not for the volitional control of the body? Is consciousness, in fact, epiphenomenal?239 Chris Frith proposes consciousness performs a radically different function to classical assumptions (2010a). Frith, who specialises in neuropsychology and social cognition, argues conscious thought facilitates social interaction through the ascription of agency to a constructed self240. “Consciousness enables us to report our experiences, thereby creating a shared mental world” (Frith, CD 2005, p. 407). Consciousness, Frith concludes, is “fundamentally social” because it affords human beings the capacity to “create group knowledge” and “optimise group decision making” (2010b).

The claim that consciousness is fundamentally social is echoed by Roy Baumeister. Rejecting the forced choice between omnipotent and impotent depictions of conscious thought, Baumeister & Masicampo favour the possibility of a third alternative: consciousness functions as an animal–culture interface.

If conscious thought is indeed useless, irrelevant, and even counterproductive for some tasks, then perhaps its adaptive value is to be found elsewhere. We look at it as an adaptation suited to the relatively sophisticated demands of the unique kinds of social life that humans develop, including culture. Many theories have assumed that conscious thought is for perceiving the environment and for directly controlling action, but the detractors have revealed its inadequacies for those functions. Instead,

167 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. we suggest that it serves the vital interface between the animal body and the cultural system and that its powers are best appreciated in terms of simulating events away from the here and now. These simulations could include replaying past events (even counterfactually) to learn from them, imagining possible courses of future action and their potential consequences, and empathically intuiting the perspectives and mental states of interaction partners. (Baumeister, Roy F. & Masicampo 2010, p. 945)

Just as human beings share the same morphology we also share the same propensity to group ourselves into complex social organisations and generate and transmit cultural knowledge through language and narrative. Indeed, narrative appears to be both automatic (Wyer, Adaval & Colcombe 2002) and universal (Barthes 1977b)241. Following Michael Tomasello (1999; 2005; 1993), Baumeister & Masicampo reason that

[h]umankind’s primate ancestors presumably had phenomenal awareness and social life. When new brain structures led to gestural communication, hominids began to share information socially, which is one foundation of culture. That became central to humankind’s biological strategy for survival and reproduction. Once vital information was held collectively, natural selection would favor individuals who were best able to access this information.

Physical evolution then changed course to facilitate social and cultural interaction. (Baumeister, Roy F. & Masicampo 2010, p. 952)

In considering consciousness from an evolutionary perspective, Baumeister is able to integrate biology and culture within the same theoretical framework242. As social engagement became more complex the unprecedented scale of information sharing laid the foundations for culture, and consciousness evolved as the body’s strategy for distributing and manipulating extensive tranches of certain kinds of knowledge stored in the individual body–minds of a population243. Evolutionary pressures selected for conscious thought as a novel strategy for survival, and thus became part of our environment. The selective environment then became both physical and social.

Human conscious thought, therefore, is an adaptive strategy which emerged as a specialised function from the phenomenal awareness of primate nonconsciousness, and evolutionary

168 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. pressures selected for social cohesion and coordination which became the foundations for culture (Baumeister, Roy F. 2005; Baumeister, Roy F. & Masicampo 2010; Baumeister, Roy F. , Masicampo & Vohs 2011). Thus, culture became part of humankind’s biological strategy for survival. The casting together of consciousness, culture and biology in an evolutionary framework recognises the principle of gene-culture co-evolution244. The essence of gene- culture co-evolution can be heard in the observation by biologist Peter Richerson and anthropologist Robert Boyd when they claim “nothing about culture makes sense except in the light of evolution” (Richerson, Peter J. & Boyd 2005a, p. 237). This puts aside the nature/nurture debate to propose modern human behaviour arises from biological and cultural processes which are now evolutionarily entwined. In effect, conscious thought – and, by extension, narrative – evolved as a tool to augment the processes of nonconsciousness (Baumeister, Roy F. & Masicampo 2010, p. 948).

At this point it is worthwhile raising a certain scholarly discontent with narrativism. Within media theory there exists a dominant claim that cinema, television and computer games are intrinsically narrative forms. Film Studies has a tendency to examine narrative cinema and the result, unsurprisingly, is the production of a conceptual framework relating to narrative. There are, however, other forms of cinematic media where non-narrative mechanisms are far more conspicuously encountered, such as music video, TVCs and computergames245. Indeed, the ‘reduced’ listening experience of recorded music reveals the power of the embodied self acting in surfeit of narrative. Not all dimensions of cinematic imagery lends itself to narrative logic246.

Such unwavering reverence for narrative, however, should not go unremarked247. Even for cognitivist film theorists like David Bordwell and Kristin Thompson classical narrative primacy is unimpeachable. Witness, for example, their defence of classical narrativity (Bordwell 2002) even as Thomas Elsaesser (2002) and Eleftheria Thanouli (2006, 2008, 2009) argue for the existence of a ‘post-classical’ cinema. Why is there such a prevailing need to acknowledge the primacy of narrative? Indeed, what now can escape the purview of narrative? As a consequence of the ‘narrative turn’ it seems a nation’s history, a corporation’s management and one’s own life may and must be conceived of in narrative terms. Galen Strawson regards this kind of deference to narrativity to be “a fallacy of our age” (2004b). Indeed, a general discontent with narrativism manifests in William F. Brewer’s ‘To assert that essentially all human knowledge and memory is represented in terms of stories is certainly

169 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. wrong’ (1995), Martin Kreiswirth’s ‘Merely telling stories? Narrative and knowledge in the human sciences’ (2000), Peter Lamarque’s ‘On not expecting too much from narrative’ (2004), and, more pointedly, Strawson’s own ‘Against narrativity’ (2004a). Dan Zahavi underscores the limits of narrative understanding when he claims that when “an experience is only appropriated as my own the moment I tell a story about it is simply wrong” (Zahavi 2007, p. 11), and Tom Kindt (2009) resists narratological expansionism when remarking “narrative is not everywhere. And, what is more, even if narrative is somewhere, it is often of sub-ordinate or even no importance to the subject at hand” (Kindt 2009, pp. 42-43). Such an arising sceptical stance toward narrativism suggests the need to bolster the theorising of other and different modes of understanding. In the context of cinematic media, I suggest this involves the mechanisms involved in perceptual design.

Undoubtedly, there are limitations to the narrative mode for generating knowledge, and it may be that the emergence of post-classical cinema has been precipitated by a collective frustration with these functional limitations. Post-classical cinema, as discussed in Chapter 2, is characterised by a marked return of a cinema-of-attractions (Strauven 2006b). After the rise of New Hollywood, mainstream cinema began to diverge into two conspicuous streams: spectacle became a strong characteristic of post-classical cinema248 and ultra-long narrative arcs became a feature of pay-television and video-on-demand networks249. The dominant trend toward intensifying both spectacle and narrative indicates a new phase of cinematic development is already underway even as it suggests an approaching phase when hyper- versions of both attractions and narrative might re-integrate to form an enhanced neo-classical form.

6.4 Summary: consciousness and cinematic media In summary, this chapter marshals neuropsychological evidence to illustrate the very deep structural relationship between the body, consciousness and narrative. The chapter builds upon theories of affect and embodied cognition reviewed in Chapter 4 to describe how the neurobiological basis of consciousness gives rise to the narrative self, storytelling, and culture. Conscious thought and the self arise from nonconscious processes as a natural consequence of embodied cognition. Narrative is a key characteristic of that conscious experience.

170 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Ample and accruing evidence discredits the notion of a narrative self in direct control of the body. While this evidence has led some researchers to claim consciousness is epiphenomenal (Bargh 1997; Pockett 2004) others have responded by proposing conscious thought to be intimately tied to culture, and propose the self to be an “animal–culture interface” (Baumeister, Roy F. & Masicampo 2010). The adaptive value of this interface is the maintenance of the well-being of the individual animal and the ensuring of species survival (Baumeister, Roy F. 2011; Baumeister, Roy F. & Masicampo 2010; Baumeister, Roy F. , Masicampo & Vohs 2011; Frith, CD 2010a).

The most profound insights from the study of consciousness for sound design and cinematic media are an understanding of the central function of affect in generating the “whirring blur” of images in core consciousness, and how this establishes narrative consciousness. Moreover, this research provides a basis for understanding the cinematic system of perceptual and narrative immersion as analogue of the embodied and narrative self. Additionally, the account alerts us to ‘story’ as a post-hoc narrative construction. These insights have significant implications for Film Studies which conventionally places greater emphasis upon the explanatory power of narrative. In the following chapter, I integrate these findings concerning narrative consciousness with the embodied simulation account of cinematic sound developed in Chapter 5 with reference to specific cinematic examples.

------this page intentionally blank ------

Truffaut once asked Hitchcock if he considered Psycho (Hitchcock 1960) to be an experimental film. Hitchcock replied:

Possibly. My main satisfaction is that the film had an effect on the audiences, and I consider that very important. I don’t care about the subject-matter; I don’t care about the acting; but I do care about the pieces of film and the photography and the soundtrack and all the technical ingredients that made the audience scream. I feel it’s tremendously satisfying for us to be able to use the cinematic art to achieve something of a mass emotion. And with Psycho we most definitely achieved this. It wasn’t a message that stirred the audiences, nor was it a great performance or their enjoyment of the novel. They were aroused by pure film. (Truffaut 1978, p. 349)

Hitchcock’s response points Truffaut to the affective primacy of “pure film” and contradicts the narrativism implicit in classical film theory. Moreover, Hitchcock equates the emotional experience of cinematic art with the perceptual properties of the moving image. I call this the perceptual design of cinematic imagery. Perceptual design challenges the tendency in standard film theory to subordinate the ‘technical’ aspects of audiovisual design to dumb carriers of narrative through the conceptual separation of content from form. In this chapter, the insights of Chapters 3, 4, 5 and 6 are put to work. It describes the structural relationship between the body and consciousness that have homologues in cinematic media as perceptual and narrative immersion. Specifically, this chapter describes the primacy of affect in cinematic media, and explores the aesthetics and embodied basis of cinematic deep structure through the concepts of perceptual and narrative design, proto-narrative, and qualities of the cinematic.

7.1 Cinematic design strategies

When you are in the middle of a story it isn't a story at all, but only a confusion; a dark roaring, a blindness, a wreckage of shattered glass and splintered wood; like a house

173 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. in a whirlwind, or else a boat crushed by the icebergs or swept over the rapids, and all aboard powerless to stop it. It's only afterwards that it becomes anything like a story at all. When you are telling it, to yourself or to someone else. – Alias Grace (Atwood 1996, p. 298)

The logic of narrative is marked by its own post-hoc construction, and Atwood’s passage captures not only the seemingly paradoxical relationship between the spectacle of sensation and story but renders it in terms similar to theories of embodied and narrative consciousness (see Chapter 6). In the cinematic context, these paradoxes manifest as the difference between the ‘being in the moment’ of perceptual immersion and the ‘understanding afterwards’ of narrative immersion250. Earlier I set out an argument for the affective primacy of cinema and proposed affective content may be produced via the strategies of perceptual and narrative design251 (see Chapters 3 and 5). Following on, the primary concern of sound design is the production and structuring of affective content through the immersive impulses of perceptual and narrative design. Of these two strategies, perceptual design is accorded prominence in this thesis because much critical analysis within Film Studies has already been concentrated upon narrative at the expense of the perceptual. In taking seriously the claim that Feeling of Body252 provides the embodied basis for narrative proper (Wojciehowski & Gallese 2011, pp. 7-8), it is possible that understanding perceptual design may be the more essential task in contemporary Film Studies.

7.1.1 Perceptual design and narrative design To reiterate, perceptual and narrative design are forms of immersion achieved by different routes. Perceptual immersion inducts core affect, mood and feeling-states within the body of the audience through (mostly) bottom-up processing while narrative immersion produces emotion and meta-emotion through (mostly) top-down processing. From a functional perspective, perceptual immersion is the simulation of physical experience (Bastiaansen, Thioux & Keysers 2009; Gazzola, Aziz-Zadeh & Keysers 2006; Keysers & Gazzola 2009; Keysers et al. 2003) while narrative immersion is the simulation of social experience (Lee 2004b, 2004a; Lombard & Ditton 1997; Mar & Oatley 2008). The primary function of these forms of immersion is to both produce and consume affect. Narrative immersion arises from perceptual immersion as a consequence of embodied simulation. From these definitions it is possible to claim the output of perceptual immersion is Feeling of Body while the output of narrative immersion is Theory of Mind.

The discussion of perceptual design, proto-narrative, and the cinematic has thus far been somewhat abstract. How useful are these terms? What does perceptual design sound, look and feel like, and how does it contrast with narrative design? Television commercials (TVCs), music video and movie trailers are media forms which regularly emphasise perceptual design over narrativity, and a rich illustration of these forms of design may be found in the teaser and trailer for Zach Snyder’s Sucker Punch (Snyder 2011c)253.

Perceptual design: Sucker Punch (2011) teaser

Figure 7.1. Perceptual design as the dominant strategy in the Sucker Punch teaser (2011).

The Sucker Punch teaser (Snyder 2011a) provides an accentuated example of perceptual design. Some of the controlling characteristics of the teaser are an extremely high density of emotion cuing and a pronounced use of audiovisual rhythm to induct affective experience. The momentum, trajectory and rhythm of moving aural and visual objects and how they shape-shift, how textures flow from one into another, primes our expectations for what might happen in the next instant. Time and space, action and things, unfold through each other. Aesthetic flow arises from perceptual causality not narrative causality.

The soundtrack of the Sucker Punch teaser is dominated by a selection of extradiegetic music cues254 and therefore does not provide an illustration of sound design in its broadest definition. Nevertheless, it does powerfully illustrate how music score operationalises a

175 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. number of BRECVEMA mechanisms for the auditory induction of affect; in particular, brain stem reflex, entrainment, contagion, musical expectancy, and aesthetic judgement (Juslin 2013b; Juslin, Barradas & Eerola 2015; Juslin, Harmat & Eerola 2013; Juslin et al. 2010; Juslin & Västfjäll 2008).

The brainstem reflex (Juslin et al. 2010, pp. 620-621) parses the environment for acoustic features such as sudden stimulus onset, loudness, dissonance and fast temporal patterns, and controls heart rate, breathing, emotional arousal, and attention. The brainstem reflex nonconsciously parses sound to produce affective response. The tempo and timbre of the music score sets an affective frame around both the visual design and the general reception of the trailer. Notably, The Crablouse (Lords of Acid 1994), which is the dominant music cue, exploits the parameters of rapid stimulus onset and fast temporal patterns, and an extreme use of timbre in the creation of affective valence and arousal.

Rhythmic entrainment (Juslin et al. 2010, p. 621) impels a listener’s internal body rhythms (such as heart rate, breathing) to synchronise to the external rhythm of a musical pulse. Moreover, the listener’s heart rate or respiration may be driven, within limits, by changes in musical pulse255. In effect, entrainment locks the listener – bodily and emotionally – into the sonic rhythm of the event. It also produces a temporal template which may be superimposed over and influence the perception of visual rhythm256. Rhythmic entrainment is capable of inducing “altered states of consciousness similar to trance or flow” (Trost et al. 2014, p. 62), and lends the Sucker Punch teaser a hypnotic unity. Additionally, rhythmic and timbral changes can induce musical chills. Chills are a phenomenon exploited in electronic dance music known as ‘the build-up’ and ‘the drop’ (Solberg 2014). Chills (sometimes also called goosebumps or piloerection ) are an autonomic peak response to affective perception, signal extreme activity in the sympathetic nervous system, and can be elicited through aural, visual and tactile stimulation as well as mental self-stimulation (Grewe et al. 2011; Salimpoor et al. 2009).

The Sucker Punch teaser contains a notable ten second sequence, occurring between roughly 00:20 and 00:30 (mm:ss), which reliably elicits aesthetic chills upon repeated viewing. These chills are the product of a confluence of audiovisual triggers framed by two music cues. The outgoing cue, Immediate Music’s Prologue to a Quest (2009), is marked by a sustained choral element which resolves with a shearing metallic crescendo. These metallic elements have a

176 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. decided quality of sound effects, and suggest the looming arrival of a large and heavy object. As the music cue resolves, it is accompanied by an exceedingly rapid visual montage of, among other things, a fireball, eyes in close-up, faces in anguish, a pistol, and motion graphics. The musical crescendo coincides with full screen text proclaiming ‘Open Your Mind’. Importantly, this moment is marked by several changes. As ‘Open Your Mind’ fades through black, the sustained choral quality of the first music cue resolves to an echoing near- silence, and, after a beat, a female voiceover calmly states “Your fight for survival …” as a samurai, large and heavy, steps forward “… starts rights now”. Orchestrated around this pause is a heavy swooshing which coincides with the samurai’s assertive shift in stance, a smaller swoosh accompanying a puff of icy breath issuing from the samurai’s mask, and another but larger swoosh which marks the start of the incoming rapid arpeggio of Lords of Acid’s The Crablouse (1994). This surge in sound coincides with a speed-ramped visual track in to the face of the crouching Babydoll (Emily Browning). What is powerful about this sequence is how the orchestration of the rhythmic changes in the perceptual design induce aesthetic chills and become targeted to Babydoll’s face257. These autonomic responses render Babydoll as both heroically defiant and vulnerable. This ten second sequence illustrates how patterns of tension and release arising from rhythmic entrainment are capable of generating core affect, and how this core affect may be focussed upon a person, object, or event.

Figures 7.2 and 7.3. Samurai (left) and Babydoll (right) prepare to fight. Perceptual design orchestrates an aesthetic chills response targeted to the face of Babydoll. Screen shots from Sucker Punch teaser (2011).

Emotional contagion (Juslin et al. 2010, p. 622) is the internal mimicry of a perceived emotion expressed by voice-like qualities inherent within a musical source. The choral arrangement of Prologue to a Quest (2009) is both ethereal and expectant, and lifts the listener to float breathlessly on air before the rapid grounding by the incoming musical rhythm.

177 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. In addition to the nonconscious automaticity of brainstem reflex, rhythmic entrainment and emotional contagion are the mechanisms of musical expectancy and aesthetic judgement. Musical expectancy inducts emotion because a “specific feature of the music violates, delays, or confirms the listener’s expectations about the continuation of the music” (Juslin et al. 2010, p. 623). While affect of this kind arises from the pleasure of pattern recognition it is also heavily influenced by social and culture milieu, as is aesthetic judgement. In this case, the aesthetic emotion derived from the teaser’s music score is heavily skewed according to the individual’s personal experience and engagement (or not) with a sub-cultural identity.

The tools of perceptual design are those techniques which energise, suppress or manipulate the multimodal mechanisms for generating affect. The BRECVEMA framework lays out the primary mechanisms for generating auditory affect, and auditory structure (as Chapter 3 reveals) can exert influence over aspects of visual perception and attention.

Within the Sucker Punch teaser, time and space flow, shimmer, segment, contract and expand. By manipulating our perception of space and time, patterns of tension and release are created. Tension possesses a fundamentally negative quality and its release possesses a positive quality. Crafting patterns of tension and release creates emotional contours in the experience of the mediawork, and these contours shape audience attention and engagement. Patterns of tension and release are able to generate what James Russell calls core affect (Russell 2003, 2005, 2009; Russell & Barrett 1999) (see Chapter 4).

Core affect is a pre-conceptual primitive process, a neurophysiological state, accessible to consciousness as a simple non-reflective feeling: feeling good or bad, feeling lethargic or energised … Its presence in consciousness varies from being focal to peripheral to out of sight. (Russell 2009, p. 1264)

Core affect is free-floating; an atom to be used in the construction of emotion proper. It may also, as a consequence of its ‘free-floating-ness’, be erroneously causally attributed258.

A key to understanding core affect is that people have no direct access to these causal connections and limited ability to track this complex causal story. Instead, a person makes attributions and interpretations of core affect. Sometimes the cause is obvious,

Russell’s concept of core affect provides a powerful way to understand how the Sucker Punch teaser may nonconsciously generate affective experience without necessarily generating narrative. It illustrates the functional difference between core and extended consciousness (see Chapter 6) by demonstrating how affect may emanate from audiovisual imagery but need not lead to an identifiable story. Indeed, the teaser deliberately eschews a sustained use of narrative causality, and is too fragmented to produce narrative absorption of any significant depth. The teaser seems less interested in character goals and motivation than in sensuous and affective access to a Feeling of Body. Within an embodied cognition account of sound design, the core affect and mood generated by the perceptual design of the Sucker Punch teaser produces what I call a proto-narrative.

Narrative design: Sucker Punch (2011) trailer

Figure 7.4. Narrative design as the dominant strategy in the Sucker Punch trailer (2011).

In contrast to the teaser stands the Sucker Punch trailer (Snyder 2011b). The trailer re-uses many of the same images as the teaser but employs a different design strategy. Where the teaser emphasises perceptual design, the trailer is concerned with narrative design. Narrative chunking and thematic cuing are made available to the audience in the form of event schema and character archetypes as well as tropes which plunder the cultural archive for

179 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. representations of gender roles and sexual identity, power and dominance, freedom and agency, struggle and resistance, through a postmodern aesthetic of ‘iconic compression’. Although brief, the trailer is capable of generating narrative absorption259. When we recognise the story’s character archetypes we gain insight into the psychological motivation and narrative function of the characters, and why they might be acting and responding in the ways that they do. That is, we build a ‘Theory of Mind’ for these characters260.

The trailer makes use of much the same imagery as the teaser, but this imagery is made subservient to the goals of an extended chain of narrative cause-and-effect. One of the parameters which both narrative and perceptual design manipulates is the perception of event boundaries and segmentation. A key characteristic of narrative design, therefore, is a specific kind of causal manipulation of event perception261. In parallel, the sensuous and affective qualities of the audiovisual imagery of the trailer invite us to feel, as Greg M. Smith (Smith, GM 2003c) suggests, and these feelings are mapped as a consequence of the free-floating nature of core affect onto the mediawork’s fictional characters to amplify narrative causality and generate social cognition.

Summary of perceptual and narrative design The perceptual and narrative design of cinematic media functions as a homologue to core and extended consciousness respectively. In describing cinematic media as an interaction between these two forms of design it might appear I am reiterating yet another form of dualism. This is not my intention, and I wish to resist Cartesianism. Indeed, an important feature of embodied cognition is the dissolution of Cartesian dualism (Varela, Thompson & Rosch 1991, p. 30). Within the EC paradigm, the mind–body split is replaced by the body–mind system262. Human consciousness arises from embodied processes. My proposed embodied simulation account of cinematic media concentrates upon perceptual design and its relationship to Feeling of Body. This might give the impression I consider narrative to be peripheral in the cinematic experience. I do not. But nor do I consider it central. Any theory of cinematic media must engage with the nature and function of story. However, embodied cognition demands enactive theories of narrative, and this suggests the traditional top-down command-and-control theories of narrative must be put aside in favour of a bottom-up embodied approach263.

Next, I examine the mechanisms of audiovisual fusion and metaphor, and develop a construal of proto-narrative as a cinematic template for affectively linking embodied schema. Proto-

180 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. narrative, as I define it, acts as an interface between perceptual and narrative immersion because it transduces sensorimotor patterns into ‘liberated’ embodied simulation (Wojciehowski & Gallese 2011, p 2). From this perspective, perceptual design is a strategy which manipulates what Lawrence Barsalou calls perceptual symbols (Barsalou 1999) and what Gallese & Lakoff call cogs (Gallese & Lakoff 2005), both of which theorise the neural reuse of the sensorimotor system in the production of abstract conceptual knowledge264.

7.2 Deep structure of cinematic media

7.2.1 Audiovisual fusion Audiovisual fusion is a form of multimodal integration (see Chapter 3), and is a basic tool for cinematic design. It arises through the feature synchronisation of aural and visual imagery. It happens rapidly, far from conscious awareness, and is instrumental in the production of presence. Audiovisual fusion is continually on display in contemporary cinema and television, but is strikingly foregrounded in the music video for Autechre’s Second Bad Vilbel (Cunningham 1996) (available on the accompanying DVD-ROM to Appendix D). Notable also is the operationalisation of the perceptual linkage between pitch and visual motion (see Chapter 3, particularly (Hedger et al. 2013)); musical glissandi are translated into visual motion, such as a scrolling frame bar or the extension of what appears to be a robotic arm.

Another notable quality is the dark and menacing mood produced by the timbres and rhythmic patterns of tension-and-release which characterise Autechre’s industrial dance music, and which are mirrored by the visual distortion designed to obscure the object under observation. There is a pervading (yet magnetic) ambiguity as to what is being witnessed. Through tight audiovisual synchronisation the music video achieves a perceptual cause and effect to lend the object agency while enigmatically eschewing narrative causality, and it may be this that gives rise to the sense of menace. An object with agency but whose intentions are unreadable registers as threatening. The world of Second Bad Vilbel, like its forebear Steamboat Willie (Iwerks 1928), is brought forth by sound and rhythm, even if that world is murky and ambivalent. Synchronisation is the most powerful tool at our disposal for creating cinematic imagery265.

Figures 7.5 and 7.6. Audiovisual fusion as a basic tool for perceptual design. Screenshots from Second Bad Vilbel (1996).

7.2.2 Audiovisual metaphor Audiovisual metaphor is an enhanced form of audiovisual fusion, and is produced by the technological recombination of sensory information. This kind of sensory recombination is a unique property of cinematic media. Unlike the mandatory integration which occurs during everyday perception, cinematic technique may substitute and recombine sensory information to generate entirely new audiovisual images. The communicative power of audiovisual metaphor is well illustrated by Engine (Cunningham 1999), a television commercial by British video artist Chris Cunningham. (Cunningham is also the director of Second Bad Vilbel examined above.) When viewed in a mute condition, the first image of Engine presents a flexing male torso. The torso is muscular, and radiates power and control, but remains contained – if not constricted – by a stationary frame which neither pans, tilts nor tracks. The flexing of arms is in plane with the chest and shoulders except for the last movement when arms thrust directly to camera whereupon a visual cut to a smooth backward tracking shot reveals a Nissan car. The colour palette and lighting is similar in both shots and consists of near-monochrome browns, greys and blacks, and an overhead but diffuse light source enhances the curves of both the male body and the body of the car. Taken together, the visual information invites a comparison between human and car. The human moves like a machine whereas the car manifests effortless grace.

When Engine is heard as well as seen the simile ‘man is like machine’ is transformed into the metaphor ‘man is machine’. The fusion of body movements and servo sounds into an audiovisual metaphor leaves no doubt concerning the reality status of the human body: the man is a machine. Moreover, accompanying the gliding reveal of the Nissan is a synthetic but flowing soundscape. In this juxtaposition of aural and visual imagery, the usual pairings are

182 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. reversed. Man is machine, car is natural. Another notable sonic quality of the TVC is the metallic shearing transition which accompanies the cut from static frame to backward tracking shot (00:11 mm:ss). The energy of the sound effect not only powers the arc of the arms as they fly up toward camera but provides the force necessary to drive across the visual cut and propel the camera backwards.

At under 30 seconds in duration, it is unclear if Cunningham’s Engine represents a story. It is difficult to say what happened or why. However, it is clear the power of the TVC emanates from an audiovisual metaphor. Moreover, in the place of narrative is a web of feelings and thoughts that illustrates Greg M. Smith’s notion of an associative network (Smith, GM 2003b, p. 29). Audiovisual metaphor recombines elements of sensorimotor schemas in novel ways to create associational linkages. An associational network, then, is a kind of assemblage of similar parts of different structures recombined to create richer structures. Audiovisual metaphors carry with them the genesis of emotion metaphors. By cross-mapping sensory features from one modality onto another, a form of synaesthesia arises and, as Richard Cytowic notes, synaesthesia is laden with affect (Cytowic 2002, p. 49) (see Chapter 3). In an associational network the affect that is associated with a particular sensorimotor schema may potentially be cross-mapped onto a different schema. This process is generative and potentially inexhaustible, and quickly permits the scaffolding of larger and larger structures. Thus, the crafting of audiovisual metaphor is a form of emotion design. Kathrin Fahlenbrach (Fahlenbrach 2002, 2005b, 2005a, 2008) has written extensively on audiovisual metaphor and its affective dimensions. Audiovisual metaphor synaesthetically maps qualities from a source onto a target domain, acting as a form of sense-making. Theories of embodied cognition consider the perception/conception distinction to be overstated (Goldstone & Barsalou 1998), and argue that abstract thinking is not purely mental and amodal but grounded in modality- specific sensorimotor processing (Trumpp et al. 2013). This view expresses a direct relationship between sensorimotor imagery and thought. Thus, the simple cinematic recombination of sounds and pictures wields the capacity to generate novel ideas, and these ideas are held in relation through their perceptual similarities and affective qualities.

Figures 7.7 and 7.8. Audiovisual metaphor of man-is-machine. Screenshots from Engine (1999).

7.2.3 Proto-narrative The concept of cinematic proto-narrative was introduced in Chapter 3 as an affective linking of the body’s experience of mediated space through time; an unfolding presence bound by an affective logic into an experiential episode. I also proposed proto-narrative to function before and below conscious thought (and, therefore, the narrative self). The purpose of this section is to develop further the concept of cinematic proto-narrative by expanding its description, providing reinforcement for the concept from other fields of study, and illustrating the concept with media examples. Lastly, proto-narrative is argued to be a product of perceptual design which interfaces with narrative design.

7.2.3.1 Definition and function of proto-narrative Proto-narrative is considered an antecedent of narrative, yet remains a somewhat ambiguous term. Variously deployed in the analysis of mother–child proto-communication (Gratier & Trevarthen 2008; Stern 1999; Trevarthen 2006, 2008), in relation to the Chinese aesthetic notion of ‘savouring’ (Sundararajan 2008), and the underlying somatic and affective basis of musical meaning (Walsh 2011), proto-narrative also bears resemblance to Silvano Arieti’s concept of ‘endocepts’ (Arieti 1976; Averill & Nunley 1992) and to what Mark Turner calls ‘small stories’ (Turner 1996, p. 13). In attempting to tighten the definition, Sundararajan observes proto-narratives to be:

‘small stories’ that are more creative than full fledged narratives, partly because of their successful resistance against the latter’s telos. By keeping the narrative impulses

184 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. to the minimum, and by resisting the temptation of the plot to “thicken,” protonarratives reduce our risk of submitting to compulsory outcomes – such as action and problem solving – of the emotion narrative. (Sundararajan 2008, p. 243, emphasis in original)

In contrast to narratives that are oriented around actions and explicit aims, protonarratives are related less to specifically goal-oriented affects and more to background moods (Sundararajan, 2000) or “endocepts” (Arieti, 1976; Averill & Nunley, 1992). Protonarratives allow ambiguity and exploration of multiple nuanced meanings and thus invite us to bypass simplistic misunderstandings of intentionality of the affective life. They are less oriented toward achieving specific goals that we value and more toward enhancing the experiencing of the value of that which we value, whatever it might be. (Sundararajan 2008, p. 244)

Proto-narratives, then, being similar to moods and endocepts, are characterised by affective states of low intensity. While moods lack specificity of object, endocepts are “amorphous cognitive-affective structures” which may nevertheless become associated with a concept or image (Sundararajan 2008, p. 246). In contrast to narrative, the logic which binds the thematic elements of proto-narrative is “devoid of temporal causal connections” (Sundararajan 2008, p. 246). Proto-narrative is instead bound together by an associative logic of affective resonance (Getz & Lubart 1998; Lubart & Getz 1997). According to Lubart & Getz, an endocept

can be described as an interconnected unit representing emotional tone, attached to a particular concept or image; the endocept is activated via the attached concept (or image) or by another endocept that matches the original endocept to some degree … endocepts may resonate with each other in memory. (Lubart & Getz 1997, p. 290)

The logic of proto-narrative is the logic of affective resonance. Affective logic of this kind affords great power because of its high malleability, flexibility, connectability and scalability. A key characteristic of proto-narrative, therefore, is an enhanced capacity to generate networks of embodied meaning.

In summary, proto-narrative is a floating non-representational affect image with no beginning or end. It stands in contrast to narrative because it is “not anchored in the temporal-causal

185 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. chain of events” (Sundararajan 2008, p. 249). Proto-narratives are “minimum units of meaning, that are gravid with multiple and nuanced emotional undercurrents” (Sundararajan 2008, p. 251) which function below the threshold of narrative. In short, proto-narrative recruits the human emotion system in the production of embodied meaning.

In the context of cinematic media, the affective logic of proto-narrative bears great similarity to Greg M. Smith’s mood-cue approach and associative network model (Smith, GM 1999, p. 108; 2003b, pp. 29-31) (see Chapter 5). In Smith’s associative network model

… the various components of the emotion system are connected by a series of associative links. Emotions (the nodes of the system) are tied to particular thoughts and memories as well as patterns of physiological reactions. Conscious cognitions (memories, social mores, emotion labels, etc.), autonomic and central nervous system patterns, action tendencies, vocalizations, and facial patterns are all interrelated. (Smith, GM 2003b, p. 29)

Highly affective, temporally fragmented, and distinctly non-narrative, proto-narrative also bears similarity to Gaudreault and Gunning’s concept of the cinema-of-attraction266:

Rather than a developing configuration of narrative, the attraction offers a jolt of pure presence, soliciting surprise, astonishment, or pure curiosity instead of following the enigmas on which narrative depends. However, this burst of presence can itself be structured by playing with or delaying its act of presentation and disappearance. Further, it can interact with narrative structures either by dominating them or by submitting to their dominance and assuming circumscribed roles within a narrative logic. (Gunning 1993, p. 10)

What attracted audiences of cinema before it was institutionalised, before it was narrativised, was the sheer phenomenal experience of technologised presence. Presence is the feeling and wonder that binds us in the immediate, and has little to do with narrative. It is, to apply Gunning’s phrase, the “smack of the instant” (Gunning 1993, p. 11).

186 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Although narrative became the dominant mode in classical cinema it never eliminated cinematic spectacle (Gunning 1993, p. 4). Indeed, as cinema moves deeper into a post- classical phase, attractions are on the ascension.

Does this mean that the cinema of attractions is the true nature of cinema and that we have to consider classical cinema as a detour or “intermezzo”? Gunning’s concept [of attractions] certainly permits to connect early cinema with avant-garde cinema, on the one hand, and pre-classical (or pre-narrative) cinema with post-classical (or post- narrative) cinema, on the other … (Strauven 2006a, pp. 24-25)

The affective and temporal profile of proto-narrative closely matches Gunning’s concept of cinematic attraction with one important difference. In my formulation of proto-narrative, bursts of presence become sequenced into an episode as a consequence of affective resonance. Affective resonance facilitates linking, layering and blurring, and permits a limited temporal extension.

Cinematic attractions are capable of functioning both independently of and within narrative, and this suggests proto-narrative could provide an affective substructure for narrative superstructure. This goes some way in responding to Gunning’s note that the “challenge of early film analysis lies in tracing the interaction of attractions and narrative organization” (Gunning 1993, p. 4). The

… capacity of an attraction to create a temporal disjunction through an excess of astonishment and display rather than the temporal unfolding essential to narrative explains one of the most interesting interactions between narrative form and the aesthetic of attractions. (Gunning 1993, p. 10)

Proto-narrative provides a conceptual tool by which to explore such an interplay of spectacle and narrative.

The spectacle of cinema-of-attractions has its roots in fairground sideshows and rollercoasters (Gunning 1986, p. 65). Indeed, the term attraction, as Gunning acknowledges, is purposefully borrowed from Sergei Eisenstein’s notion of “montage of attractions” (Eisenstein 1947 [1923]):

In his [Eisenstein’s] search for the ‘unit of impression’ of theatrical art, the foundation of an analysis which would undermine realistic representational theater, Eisenstein hit upon the term ‘attraction.’ An attraction aggressively subjected the spectator to ‘sensual or psychological impact.’ According to Eisenstein, theater should consist of a montage of such attractions, creating a relation to the spectator entirely different from his absorption in ‘illusory imitativeness.’ I pick up this term partly to underscore the relation to the spectator that this later avant-garde practice shares with early cinema: that of exhibitionist confrontation rather than diegetic absorption … Then as now, the ‘attraction’ was a term of the fairground, and for Eisenstein and his friend Yuketvich it primarily represented their favorite fairground attraction, the roller coaster … (Gunning 1986, p. 66)

Counterintuitively, a ride on a rollercoaster can provide an instrument by which to pry apart the tangle of the cinematic processes to reveal the function of proto-narrative in recruiting the human sensory and emotion systems when processing motion into meaning.

7.2.3.2 Proto-narrative as experiential template The cinematic was defined in Chapter 3 as being rooted in the relationship between movement and affect. In this section, a rollercoaster ride, when stripped to its basic components, presents a prototypical cinematic experience, and provides a tool by which to explore the underlying kinaesthetic, proprioceptive and affective mechanisms of cinema without considerations of story267.

Rollercoaster as proto-narrative The definition of an emotional roller coaster (Thatcher 2009) is a home video posted to YouTube268. It documents a boy’s first encounter with a particular themepark ride. At first, there is something raw and confronting about the video. We know it to be nonfictional and are uncomfortable witnessing a boy overwhelmed by fear. Moreover, we are unsettled by the continued recording despite the boy’s terror. Both these responses are high in meta-emotional content for the audience. That is, we feel discomfort and maybe even guilt or shame at viewing a subject who is so obviously distraught and emotionally vulnerable. For the purposes of analysis, however, it is important to recognise a distinction between the emotions and meta-emotions elicited within us as an audience of the video and the affect experienced

188 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. by the boy as documented by the video. It is the latter distinction, video-as-documentation, that I wish to focus on so that proto-narrative may be revealed.

The definition of an emotional roller coaster (2009) is barely two minutes and twenty seconds in duration but provides a unique opportunity to explore the affective experience of a body moving through space. The YouTube video documents a boy propelled along a rollercoaster and the sensations of momentum and centrifugal force this movement generates. Changes in physical force elicit kinaesthetic experience and register as display markers on the boy’s face, voice and body. Each marker signals the onset of new feeling-states – an affective beat – and mirrors the boy’s kinaesthetic state as he moves through physical space (including the dimensions of acceleration and deceleration).

Figures 7.9–7.12. The experience of hurtling through space structured by anticipation, apprehension, and peak moments of terror and joy. Screenshots from The definition of an emotional roller coaster (2009).

The markers chart a logic and sequence of affect which unfolds through time. Figure 7.13 describes the sequence and rates affective intensity as either increasing or decreasing in tension269.

Marker Marker description Intensity (tension)

1 Interest Increasing tension

2 Anticipation Increasing tension

3 Apprehension Increasing tension

4 Fear Crisis (tension)

5 Terror Crisis (peak tension)

6 Coping Decreasing tension

7 Exhilaration Decreasing tension

8 Joy Decreasing tension

9 Apprehension Increasing tension

10 Fear Crisis (minor)

11 Coping Decreasing tension

12 Joy Decreasing tension

13 Happiness Decreasing tension

14 Satisfaction Decreasing tension

15 Anticipation (repeat of experience) Coda

Figure 7.13. Affective structure of The definition of an emotional roller coaster (2009).

Markers 1–3 (interest, anticipation, apprehension) may be grouped together under the general label of rising tension while markers 4 and 5 (fear, terror) are brief, highly negative and function as a form of crisis270. Markers 6 to 8 are also affectively charged but with positive valency (coping, exhilaration, joy). Markers 9 to 14 (apprehension, fear, coping, joy,

190 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. happiness, satisfaction) are a rapid but less intense redux of markers 3–8, and the experience flows with decreasing intensity from the point of secondary crisis to arrive at an arousal state somewhere above the baseline from which the experiential curve began271. Marker 15 indicates the end of the episode and an appetite for a repeat of the entertainment experience.

Figure 7.14. Rollercoaster as proto-narrative. Emotion markers mapped against time and intensity.

The markers act as affective beats which may be graphically mapped as tension versus time (Figure 7.14) resulting in a curve which displays unity and wholeness. The experience has a beginning, a middle, and an end. It describes the emotional logic of an experience as it unfolds through time. I take this contour to be a prototypical structure which compresses the affective spatio-temporal qualities of the rollercoaster into an experiential template272. The experiential template is the proto-narrative. Proto-narrative is the embodied meaning of the feeling of movement.

‘Affective beat’ not ‘story beat’ It is important to differentiate the term ‘affective beat’ (which manifests in the perceptual flow of cinematic imagery) and the concept of ‘story beat’. Robert McKee, who leans heavily upon the Aristotelian tradition in his teachings on screenwriting, considers the story beat to be the most basic unit of narrative (McKee 1998, p. 37). McKee’s influence in screenwriting also

191 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. reaches into interactive media. Michael Mateas and Andrew Stern, developers of the acclaimed interactive narrative Façade273 (Mateas & Stern 2005b), make explicit reference to McKee’s ideas when discussing the game’s AI-based ‘drama manager’:

Scenes are composed of beats, the smallest unit of value change. Any activity below the level of the beat is not associated with value change. Roughly, a beat consists of an action/reaction pair between characters. For example, in the case where action is being carried by dialog, a beat could simply consist of one character speaking a line of dialog, and another character reacting. (Mateas & Stern 2002, p. 4)

A story beat, therefore, is a unit of social interaction, and nothing in story terms exists below this level. From a storytelling perspective this claim seems self-evident but is deeply problematic from the perspective of embodied cognition. Mateas and Stern associate a story beat with a “value change”.

A value is a property of an individual or relationship, such as trust, love, hope (or hopelessness), etc. In fact, a story event is precisely any activity that turns a value. If there is activity – characters running around, lots of witty dialog, buildings and bridges exploding, and so on – but this activity is not turning a value, then there is no story event, no dramatic action. (Mateas & Stern 2002, p. 4)

It is here the conceptual gap between narrative and proto-narrative is laid bare. Aristotelian dramaturgical theory quarantines meaning at the level of social interaction and ‘everyday emotions’. Everyday emotions (such as trust, love, hope) are folk-psychological categories used in the commonsense evaluation and prediction of human behaviour. However, folk- psychological categories of emotion are troublingly fuzzy (Fehr & Russell 1994) and tend to be poor categories for scientific study (Russell 2012)274. As discussed in Chapters 4 and 6, emotions arise from core affective processing, processing also fundamental to the narrative of human consciousness. This is significant in the instance of cinematic media for core affect can be induced by ambient qualities in the perceptual environment, and “can exist without being labeled, interpreted, or attributed to any cause” (Russell 2003, p. 148) thus eluding capture by the notion of a ‘story beat’. A ‘story beat’, then, from the perspective of the affective sciences is a folk-psychological term which is not, as McKee claims, the “smallest

192 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. element of [narrative] structure” (McKee 1998, p. 37) because it is itself constructed from core affect. The story beat is insensible to the fine-grain of affective pulses which comprise it.

An affective beat is produced by a shift in core affective value derived from a change in internal or external stimulus. I equate affective beat with Greg M. Smith’s concept of an emotion cue (see Chapter 5). According to Smith, an emotion cue is a packet of perceptual information which activates the emotion system (Smith, GM 2003b, pp. 29 and 42). An emotion cue is

the smallest unit for analyzing a text’s emotional appeals. Emotion cues are the building blocks that are used to create the larger structures such as emotion markers. Mood is sustained by a succession of cues, some of which are organized into larger structures (narrative obstacles, emotion markers), some of which are not. (Smith, GM 2003b, p. 47)

Hence, the more densely cued the flow of audiovisual imagery, the more likely an experience of emotion will occur. Curiously, Smith implies the two parallel (but entwined) structures of proto-narrative and narrative when he observes “[a]lmost every narratively significant moment has the potential to provide some emotional payoff, but not every emotional payoff is narratively significant” because the “emotion marker is not there simply to advance or retard the narrative’s progress” (Smith, GM 2003b, p. 44). Classical narratology cannot readily explain the function of core affect within cinematic media, but it is this difference in affective function which is at the heart of the distinction between proto-narrative and narrative.

Proto-narrative as interest curve The concept of proto-narrative is equatable with what Jesse Schell terms an “interest curve” (Schell 2008a) (see Figure 7.15). Schell’s background in rollercoaster and computergame design275 identifies an interest curve as an affective structure which arises from the experience of a body moving through space (either real or virtual), and is marked, unsurprisingly, by phases similar to the experiential curve of The definition of an emotional roller coaster (Thatcher 2009) (see above).

Figure 7.15. Entertainment experiences possess an ‘interest curve’. Adapted from Schell (2008b, p. 248).

Schell’s insight concerning the affective dimensions of movement through real and virtual space echo Sheets-Johnson’s argument for the ‘primacy of movement’ (Sheets-Johnstone 1999b, 1999a), and suggests interest curves to be the germ of sense-making.

Figure 7.16. Freytag’s Pyramid of dramatic structure.

There is also a striking resemblance between proto-narrative as it is expressed though the experience of a rollercoaster and the classical Aristotelian dramatic arc of rising and falling tension described by Gustav Freytag’s ‘pyramid’276 (Freytag 1863) (see Figure 7.16). Both structures are affective in nature, but proto-narrative is concerned with the body’s kinaesthetic response to the physical forces of movement while Freytag’s Pyramid is concerned with the social forces which drive narrative action. Proto-narrative is concerned with perceptual causality while story is concerned with narrative causality indicating that proto-narrative is functional in generating Feeling of Body while narrativity generates Theory of Mind.

The concept of tension-and-release is readily applicable to music cognition. The nature and function of musical tension has been developed by Frank Lerdahl and Carol Krumhansl (Krumhansl 2002; Krumhansl & Lerdahl 2011; Lerdahl & Krumhansl 2007) who not only observe “the ebb and flow of tension is basic to the musical experience” but that musical tension “can be induced by many factors, such as rhythm, tempo, dynamics, gesture, and textural density”. Moreover, Vines, Nuzzo & Levitin propose musical tension to possess a cyclical dimension277:

Given that tension increases and decreases throughout a musical piece, we can introduce an analogy in which fluctuations in tension correlate with an ebb and flow of affective (or musical) energy that is stored up and released within a listener. Increasing tension in the piece generates a buildup of affective energy in the listener; a subsequent decrease in tension results in a release of the pent-up energy. (In this way, the listener functions like a capacitor by storing up a charge until a threshold is reached, at which point the charge is released.) The storage and release of affective energy causes changes to a listener’s experience of emotional content … stored musical energy has the potential to alter the emotional state of a listener when it is ultimately released. (Vines, Nuzzo & Levitin 2005, p. 140)

Associated with musical tension are the concepts of rhythm, predictability, expectancy and suspense, and these concepts have their counterparts in cinematic structure278 which strongly suggests the interest curves of proto-narrative are transmedial phenomena.

195 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Once you start thinking about games and entertainment experiences in terms of interest curves, you start seeing the pattern of the good interest curve everywhere. You can see it in the three-act structure of a Hollywood movie. You can see it in the structure of popular songs. … When Aristotle says that every tragedy has a complication and a denouement, you can see it there. When comedians talk about the ‘rule of three’, you can see the interest curve. … You can even see the pattern quite concretely in the layout of a rollercoaster track. (Schell 2008b, pp. 249-250).

Proto-narrative possesses a fundamental, robust, and deeply satisfying structure. Proto- narrative is not narrative and yet resides at the heart of narrative experience. Proto-narrative appears to be an omnibus for a wide variety of experiences and, once mobilised, provides the foundation upon which narrative may be scaffolded. Proto-narrative, therefore, may be argued to be the embodied and affective anchor of narrative.

Proto-narrative is fractal Interest curves also appear to be fractal (Schell 2008b, pp. 251-252). That is, each affective curve has “an internal structure that looks like the overall pattern” (Schell 2008b, p. 252). The fundamental structure is highly scalable, and an interest curve is likely to be as robust at the micro level of a 30-second TVC as at the macro level of a 10-part season of serial television drama.

Figure 7.17. Interest curves are fractal. Adapted from Schell (2008b, p. 252).

Proto-narrative resides deep within the heart of narrative because, as Gunning reminds us, “attractions are not abolished by the classical [narrative] paradigm, they simply find their

196 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. place within it” (Gunning 1993, p. 4). A strong example of this function is the ‘Tokyo Tonight’279 sequence of Babel (Inárritu 2006)which moves aesthetically from documentary realism to techno-expressionism and back to an observant realism. The peak of the sequence occurs at a dance club where the meshing and unmeshing of audiovisual and narrative rhythms overwhelm the protagonist, Chieko (Rinko Kikuchi).

Figures 7.18 and 7.19. Entrainment and the meshing/unmeshing of audiovisual rhythm to both represent and induce peak aesthetic experience. Screenshots from Babel (2006).

‘Blockbuster’ movies tend to be ‘rollercoaster’ movies, and the fractal nature of proto- narrative is readily identifiable throughout the genre280. One such blockbuster is Peter Jackson’s remake of King Kong (Jackson 2005). King Kong provides illustration par excellence of the fractal and massively scalable nature of proto-narrative in contemporary cinematic spectacle. In Jackson’s remake, a pivotal sequence pits the central characters of Ann Darrow (Naomi Watts) and Kong against a slew of antagonists281. In a clear and pronounced way, the sequence illustrates how proto-narrative operates fractally282.

The sequence ups the ante of the original by Cooper & Schoedsack (1933). In the 1933 version the sequence runs an approximate duration of 4 minutes while in Jackson’s version the sequence lasts in the order of 11 minutes. It is not, however, merely in duration where the two differ. Aural and visual space is used entirely differently. In the 1933 version both the aural and visual frames are static and planar. In the 2005 version the frames are highly dynamic, and continually expanding and contracting. Objects approach, recede and rotate rapidly and complexly through space. Point-of-audition (POA) and point-of-view (POV) surge through cinematic space. Where the proscenium arch is preserved in the 1933 version it is exploded in the 2005 version by perceptual design techniques for generating spatial

197 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. immersion283. The sophisticated spatial dynamics of this form of design underpin meaning at both local and global levels284.

Figures 7.20–7.50. T-rex battle sequence illustrating the fractal nature of proto-narrative. High density emotional cuing distorts cinematic space and time. Screenshots from King Kong (2005).

Compared to its predecessor, the audiovisual design of Jackson’s King Kong exhibits massively enhanced affective cuing. Smith’s concepts of emotion cues and markers are on full display during this sequence, and the sheer density of affective cuing (particularly for fear) produces distortion in the audience’s perception of time. The T-rex battle is an intense set- piece of contemporary cinematic techniques developed to produce cinematic immersion and

200 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. embodiment, and a jolting illustration of how sensory information cues affective perception to orient a person to their environment.

7.2.3.3 Music video as proto-narrative Music video provides unique possibilities to isolate and analyse the qualities and function of proto-narrative because narrative is often nonexistent or immaterial in these forms of cinematic media. A central function of music video is the communication of emotional experience, and the form readily illustrates how BRECVEMA mechanisms and audiovisual rhythm induct and structure affect, and how embodied meaning arises as a result without recourse to narrative comprehension. Although the genre is not bereft of narrative examples, the embodied processes for sense-making do not require narrative for success. Narrative, in this instance, is deferential to cinematic attractions. Moreover, because the visual imagery of music video is designed post-hoc, the usual production process of filmmaking is reversed. In music video, claims of visual primacy are rendered invalid because the visual image is created too late in the production process to influence the structure of the soundtrack. Visual structure is determined by musical structure. Music video stands in contrast to classical cinema because it exemplifies a genre that privileges neither visual nor narrative primacy. As a consequence, music video serves as a laboratory where the solipsistic theorising of classical film-theory is set aside to allow a closer analysis of cinematic processes. Music video, therefore, provides fertile space for studying how crossmodal interaction inducts affect, generates novel imagery, and shapes attention.

Star Guitar (Gondry 2001b) is a high-profile music video by Michel Gondry of an equally high-profile music track of the same name by electronic dance music duo The Chemical Brothers285. Like many music videos, Star Guitar produces a distinctly non-narrative example of cinematic affect, and provides an excellent illustration of the concept of proto-narrative where the “yoke of the narrative telos is broken” because it is “not anchored in the temporal- causal chain of events” (Sundararajan 2008, p. 249).

Star Guitar is ostensibly a single shot which visually documents a train journey through industrial, rural and urban landscapes. The journey has waypoints but no destination. It does not leave anywhere nor does it arrive anywhere. Along the way the visual flow loops, stutters, dissolves and shimmers but the flow never stops. Star Guitar is journey-as-flow but not journey-as-story nor is it metaphoric for ‘story’. Gondry’s music video is not interested in

201 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. telling a story nor does it contain characters in the traditional sense. If there are characters they are glimpsed only as reflections in windows, presumably fellow travellers like us, the cinematic ‘I’. Certainly, the music video contains no specific protagonist, no antagonist, and no dilemmas are confronted. Star Guitar displays no narrative chain of events. If there is causation it lies in the ability of the aural to bring forth the visual.

A central design feature of Star Guitar is its application of the concept of sound sampling to the production of moving pictures. Buildings, storage tanks, gantries, factories, smokestacks, bridges and railway cuttings are ‘sampled’ as ‘visual objects’ and re-introduced into the visual flow of the journey to create novel compositions286. Star Guitar cuts and pastes visual objects through digital compositing to eschew montage for collage. Moreover, these visual objects are linked to specific psychoacoustic effects and musical elements so as to appear, move and disappear in rhythmic synchronisation to the beat structure of the music. In the stead of narrative complication there are changes in audiovisual rhythm to produce difference.

Figures 7.51–7.58. Visual objects are cut-and-pasted and slaved to sonic structure. Screenshots from Star Guitar (2001).

Sound slaves vision Like most music videos, Star Guitar inverts the conventional audiovisual hierarchy of live- action filmmaking. Visual production is slaved to sound because of the pre-existence of the music track. It is the music of Star Guitar which warps and wefts the landscape to bring forth patterns of gantries, smokestacks and bridges. Narrative is lacking yet meaning emerges from audiovisual rhythm. Rhythm animates the world, and the music video visualises the spatiotemporal structure of that world287.

Gondry’s technique of rhythmic audiovisual repetition is re-worked in his music video of Kylie Minogue’s Come into My World (Gondry 2002) which substitutes linearity for circularity288. The circular motion of the camera and the layered repetition of Minogue in ever-complicating situations creates something akin to a cause-and-effect chain which casts

203 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Come into My World as oddly quasi narrative289, and seems to bias toward a ‘social’ interpretation of the action.

Figures 7.59–7.64. Layered repetition creates a quasi-narrative. Screenshots from Come into My World (2002).

204 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Gondry’s visual-object-as-sound-object technique is also found in his music video for Daft Punk’s Around the World (Gondry 1997). Around the World, a forerunner to both Star Guitar and Come into My World, features a choreography driven by a mathematical progression of visual objects (in this instance, clusters of human bodies) to represent the movement of musical objects290. What sets Star Guitar and Come into My World apart from Around the World is their manipulation of temporality. Star Guitar folds space through time, Come into My World layers space over time, but Around the World progresses linear space through time291.

7.2.3.4 Evolution, proto-narrative and narrative In summary, proto-narrative is a basic and prototypical form of mediated experience. In the context of cinematic media, proto-narrative is a sense-making process which draws upon the power and flexibility of the human sensorimotor and emotion systems to process audiovisual imagery. This imagery is nonconsciously assigned core affective value to produce embodied meaning through the deep interconnection of the sensorimotor and emotion systems (see Chapter 4). My definition of proto-narrative promotes it as an affective spatiotemporal template292 which constitutes the embodied meaning of an unfolding perceptual experience, and so points to what Mark Johnson terms an “embodied schema” (Johnson, M 1987). In the context of cinematic media, proto-narrative is a sense-making process capable of creating scalable affective structures from a repertoire of embodied schema. Proto-narrative is the body’s understanding of the flow of life before it is narrativised.

In this section, proto-narrative has been exemplified as ubiquitous and transmedial. From rollercoasters and computergames to music video and the spectacle of cinematic blockbusters, proto-narrative appears in many guises. Driven by bottom-up processes, proto-narrative has a biological basis, and this suggests it piggybacks upon evolutionary adaptation. The 4E research programme claims embodied mechanisms underpin human cognition and consciousness. From an evolutionary perspective, the body–mind emerged as a consequence of selection pressures associated with moving through and interacting with a physical environment (see Chapter 4). Social interaction and intersubjectivity emerged subsequently from the capacities of the embodied mind; that is, using Jeannerod’s nomenclature, the narrative self emerged from the embodied self.

205 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Is it possible to claim art and narrative to be evolutionary adaptations?293 Katja Mellmann (2012, p. 31) cautiously proposes this could be the case while acknowledging the difficulties in empirically testing how narrative may be a true biological adaptation294. Similarly, confronting the explanatory origins of storytelling as a choice between evolutionary adaptation or unintended by-product, Michelle Scalise Sugiyama295 leans toward evolutionary adaptation (Scalise Sugiyama 2001a, 2001b, 2005, 2008). In media psychology, following Steen & Owens (Steen & Owens 2001), Ed Tan (Tan, ESH 2008) takes entertainment to be an adaptive activity. As Tan explains, entertainment is the proximal cause for distal function:

The proximal or immediate cause of engaging in entertainment is a wired-in predisposition for exploration, challenge, and play that is released by, and a response to, boredom. The distal cause is the evolved function of organization of various adaptive activities and states of mind. … People appear to engage in entertainment activity because nature has designed them to love it individually under the right conditions (proximal cause), but without being aware that they are obeying an adaptive principle that goes for the species as a whole, which is to train useful capabilities (distal cause). To give an example, the proximal causes of a child engaging in chase play may be a conscious break from routine, and ‘just feeling like it,’ whereas the distal cause is a need she is not aware of to hone her skills of dealing with aggressors. (Tan, ESH 2008, p. 31)

If the distal cause of entertainment is the need for training, what is the adaptive value of a rollercoaster or King Kong’s T-rex battle? One solution may be found in the affective logic of proto-narrative. In The definition of an emotional roller coaster (Thatcher 2009), at the apex of the affective curve (crisis), the beats flow rapidly from fear > terror > coping > exhilaration > joy (see Figure 7.13). What is pivotal is the beat of coping which functions as the linchpin between terror and exhilaration. Because the emotion-episode is framed as entertainment and not a life-or-death struggle, the experience is imbued with play and safety. One may experience and learn to cope with fear without suffering the usual consequences of hurtling at high speed toward the ground. The intense transition from negative to positive valency offers the opportunity for exercising emotional regulation and learning296.

There is also another avenue to consider regarding the evolutionary function of narrative. A characteristic development in contemporary cinematic entertainment is the impulse toward

206 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. longer and more complex narrative works. Where movies and tele-series were once limited to narratives of modest scope and duration, offerings from pay-television and streaming services such as HBO and Netflix reveal the general public is both capable and willing to follow exceedingly long and complicated multi-threaded narrative arcs. Works indicative of multi- narrative cinema are Short Cuts297 (Altman 1993), Magnolia (Anderson, PT 1999), Timecode (Figgis 2000), Adaptation (Jonze 2002a), Love Actually (Curtis 2003), Crash (Haggis 2005), Babel (Inárritu 2006) and Cloud Atlas (Tykwer, Wachowski & Wachowski 2012) to name only a handful of examples. Indicative of multi-narrative television are The Sopranos (The Sopranos 1999–2007), 24 (24 2001–2010), The Wire (The Wire 2002–2008), Deadwood (Deadwood 2004–2006), Mad Men (Mad Men 2007–2015), and Game of Thrones (Game of Thrones 2011– ). Game of Thrones, in particular, is a strong illustration of how contemporary audiences are not only prepared to engage deeply with storyworlds populated by a vast number of characters but actively hunger for them298.

What can explain the impulse toward longer and more complicated narratives? From the perspective of evolutionary and media psychology, art and entertainment may be theorised as forms of play299 which enhance learning and unconsciously rehearse adaptive capacities (Ohler & Nieding 2006b, 2006a; Steen & Owens 2001; Tan, ESH 2008; Tooby & Cosmides 2001; Vorderer 2001). What links distal and proximal causation in this model of entertainment is the experience of emotion, firstly, for the sheer enjoyment of the entertainment episode in its own right and, secondly, as an unconscious driver for training (Tan, ESH 2008, pp. 32-35). On this account, the proximal function of engaging with narrative media is to playfully (and safely) experience a broad spectrum of basic and social emotions while the distal function is to expand the repertoire of problem-solving strategies to aid navigation of an increasingly complex social environment. From the perspective of evolutionary psychology, narrative can influence the self-concept (Richter, Appel & Calio 2013), and dense narrative works can be said to function like cognitive tools in the re- engineering of an individual’s cognitive architecture (Wilson, M 2010, pp. 182-184). Margaret Wilson describes how “fundamental processes of daily cognitive activity involve the operation of cognitive tools that are not genetically determined but instead are invented and culturally transmitted” (2010, p. 180). It is likely, cinematic media function as cognitive tools, and thus possess similar capacities.

207 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. On a more speculative note, it may be the neural mechanisms which underpin conscious thought are being expanded and rendered more efficient through repeated exposure to evermore complex narrative works300. A consequence of this exposure could be an enhancement of the human capacity for planning and organisation and increased cultural sophistication, all of which are the hallmarks of our biocultural strategy for survival. If this is correct, popular culture may be performing a highly adaptive function301. Remember, Damásio’s extended consciousness – that level of autonoetic consciousness where the autobiographical self and full-blown narrative understanding occurs – requires access to large amounts of episodic, working, and short- and long-term memory. Increasing complexity of narrative works may exert evolutionary pressure upon these forms of memory and thus expand the capacities of extended consciousness.

Cinematic media are composed of a productive tension between the tendencies of attractions and narrative. My use of the phrase ‘productive tension’ is intentional, and refers to Hans Gumbrecht’s observation of the “productive tension between meaning effects and presence effects” (Gumbrecht 2004, p. 19). Although Gumbrecht’s phrase appears to connote antagonistic forces, he explains them as complementary302. Here I parallel the tension Gumbrecht identifies between ‘presence effects’ and ‘meaning effects’ and the tension expressed within Film Studies between spectacle and narrative respectively. The precise mechanisms by which spectacle and narrative interact remain unclear but, as this section argues using theories of embodied cognition, proto-narrative performs the function of an interface between the ‘presence effects’ of perceptual immersion and the ‘meaning effects’ of narrative immersion. Theories of embodied cognition suggest social interaction and intersubjectivity are underpinned by processes of embodied simulation. Proto-narrative suggests itself to be a tool for liberated embodied simulation and the production of Feeling of Body, and reveals the deep biological basis of cinematic media.

7.3 Body as basis for knowledge In Chapter 3 I described cinema as an affective spatio-temporal system. What makes cinema ‘cinematic’ is the affective content generated in response to moving imagery. In this section, and at the risk of introducing yet more neologisms into a fractious field303, I expand upon the concept of the cinematic to explain how moving imagery creates affective content at an embodied level.

The ‘cinematic’ The cinematic is the affective dimension of mediated sensations arising from a moving image304. Cinematic qualities are predominantly affective qualities, and are induced through the mediated experience of space, time and movement. Affect of this kind is not generated in response to narrative305. ‘Feeling is first’306, Zajonc reminds us, because “the very first stage of the organism's reaction to stimuli … are affective” (Zajonc 1980, p. 154). Affect is basic, primary, inescapable, irrevocable, and difficult to verbalise (Zajonc 1980). Affect is fundamental to the sense-making process which underpins cognition and consciousness.

[T]he deep structures that generate conscious state are not only responsible for the level but also for the core quality of subjective being. Conscious states are inherently affective … Consciousness is not generated in the cortex; it is generated in the brainstem. Moreover, consciousness is not inherently perceptual; it is inherently affective. (Solms 2013, p. 12, emphasis in orginal)

The cinematic arises not merely from moving pictures but moving imagery. Moving imagery, of course, puts us in contact with theories of mental imagery. Theories of mental imagery are as highly contested as the Zajonc-Lazarus debate concerning emotion and cognition (discussed in Chapter 4). The mental imagery debate is characterised by opposing accounts developed by Zenon Pylyshyn (Pylyshyn 1973, 1981, 1999, 2002, 2003) and Stephen Kosslyn (Kosslyn 1980, 1981, 1994, 2005; Kosslyn, Ganis & Thompson 2003). For the sake of transparency, I follow Kosslyn in this debate when he asserts mental imagery to be depictive and analogue to space307. Depictive theories suggest mental imagery to be a form of simulation which draws upon the same neural mechanisms as perception. As Kosslyn, Ganis & Thompson claim:

… most of the neural processes that underlie like-modality perception are also used in imagery; and imagery, in many ways, can ‘stand in’ for (re-present, if you will) a perceptual stimulus or situation. Imagery not only engages the motor system, but also affects the body, much as can actual perceptual experience. (Kosslyn, Ganis & Thompson 2001, p. 641)

209 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. The cinematic is intensely multimodal, dynamic, affective, and synaesthetic. Cinematic media, therefore, are those media capable of communicating affective moving imagery. There was once a time when film alone was emblematic of a cinematic medium. This has ceased to be the case. What is cinematic now leaks across boundaries, and is routinely encountered in media as diverse as television, computergames, virtual reality (VR), graphic novels, and music308. It has been claimed “a movie that consists of sound and no image” cannot be cinema (Cumbow & Johnson 1985, p. 64) and yet cinematic qualities are precisely what we find within contemporary music. ‘Cinematic music’ has been applied as a genre label to trip-hop group Portishead (McGuire), and the music of Massive Attack is described as “a darkly sensual and cinematic fusion of hip-hop rhythms, soulful melodies, dub grooves, and choice samples” (Ankeny 2017). How is it the experience of music may be considered to be cinematic?

The BRECVEMA framework, particularly the mechanism for the induction of visual imagery, provides an explanation as to how music may be deemed ‘cinematic’. Drawing upon Zoltán Kövecses’ work in emotion metaphors (2000) and Lakoff & Johnson’s work in cognitive linguistics (1980a), Juslin suggests “musical structure [is conceptualised] in terms of a nonverbal mapping between the metaphorical ‘affordances’ of the music and ‘image- schemata’, grounded in bodily experience” (Juslin 2013b, p. 242). Moreover, Istvan Molnar- Szakacs and Katie Overy argue the MNS may link “music perception, cognition and emotion via an experiential rather than a representational mechanism” (2006, p. 239), and this suggests how sound, motion and emotion may coalesce as affective imagery. Taken together, these insights return us to the explanatory paradigm of embodied cognition and embodied simulation theory.

7.4 In the Cut (2003): an embodied simulation approach

7.4.1 Cinematic imagery, image schemas, and conceptual thought How may an embodied simulation account assist in explaining how a cinematic work creates meaning? I shall illustrate by examining several sound design tactics used in Jane Campion’s In the Cut (Campion 2003), limiting my discussion to only those elements for which I was directly responsible or responsible in conjunction with fellow sound designer Peter Miller309. My intention here is to make explicit some of the design factors with which we were

210 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. experimenting; in effect, I am pointing to the kind of practice-based research which is typical in the industrial context of movie-making.

Pertinent to an embodied simulation approach are the terms emotion cue, emotion marker and associative network garnered from Smith’s concept of cinematic mood-cueing. To recap, an emotion cue (Smith, GM 2003b, pp. 29 and 42) is a packet of perceptual information which activates the emotion system. Emotion cues are, on Smith’s account, basic units of cinematic affect, and may be coalesced into emotion markers. Emotion markers are clearly discernible textual cues for eliciting emotion (Smith, GM 2003b, p. 44), and function as nodes which interconnect the body’s emotional subsystems to activate associative networks (Smith, GM 2003b, pp. 29-31). Associative networks underpin an affective ecology such that the experience of mood increases the likelihood of experiencing an emotion proper which further sustains (or alters) mood (Smith, GM 2003b, p. 42). I suggested core affect is functionally implicated in Smith’s notion of an emotion cue, and so may be causally misattributed within the diegesis. It is causal attribution of this kind which may be leveraged during the design of cinematic imagery to shape the perception of characters, objects, environments and scenarios before becoming absorbed into the diegesis. I also proposed that if emotion markers are sufficiently densely cued and coordinated, they may form into proto-narrative; but if the cuing falls below threshold, the core affect remains unconsolidated as mood.

Smith suggests an emotion cue is the “smallest unit for analyzing a text’s emotional appeals” and may arise through “narrative situation, facial and body information, music, sound, mise- en-scène, lighting” (Smith, GM 2003b, p. 47). I submit that emotion cues are composed of core affect, and are more granular than Smith suggests. For example, to exemplify an emotion marker, consider an audiovisual rendering of a speeding motorcycle. The marker is a composite of cues such as motion blur and the glint of chrome, the whine of the motorcycle’s engine, the tearing of its tyres on asphalt, the Doppler effect as the motorcycle passes, and the thrumming echo from nearby building walls. Notice how the single event – a motorcycle speeding past – may be dissociated into the smaller and smaller aural and visual components. Each of these aural elements may be decomposed into smaller and smaller sub-elements (shifting rhythm and timbre of engine, gears and tyres, rapidly rising and falling loudness, the spatialising psychoacoustic effects of Doppler and echo) thus acting as vectors for generating core affect and mood. The more finely detailed the sound, the more likely affect will be inducted. When edited and mixed, these cues fuse into an emotion marker.

The nature of this process bears repeating because it is a central driver powering modern practices of cinematic sound design. The technique and purpose of layering sound fragments to form affective events is very clearly described by Richard King in an interview where he discusses the sound design of a scene in Inception where psychic forces erupt around a Parisian street cafe (Coleman 2011, 01:52-03:10 mm:ss). King speaks to three distinct layers which structure the sound effects and atmospheres310. The first is a literal layer of eruption and debris surrounding the main characters, Cobb (Leonardo DiCaprio) and Ariadne (Ellen Page). The second is a layer of extreme bass and subsonic effects which, as King describes, are the “subtext sound, the energy that is causing all of this to happen” (Coleman 2011, at 2min:35sec). The third is a layer of organic sounds, such as whale calls, which have been re- pitched and manipulated. During the interview, King plays each set of sounds against the visual scene and, for clarification, inserts the DAW screen to illustrate the layered structure of the sound effects (see Figures 7.65–7.67). This is repeated for each of the three layers of sound to reveal how the scene is composed of many, many aural fragments, how these fragments are conceived in terms of timbre, and how they are aggregated in time to build larger sonic events. The purpose of this design, as King plainly states, is to make the scene frightening.

Figures 7.65–7.67. Sound fragments grouped and layered to create large-scale sonic events, emotion cues and markers. Screenshots from The sound of Inception (Coleman, 2011).

The strategy of designed multi-component sound is not confined to big-budget spectacle movies. The same process is used in most feature cinema and television drama, even documentaries, and it is used in In the Cut. Returning to my own work, a feature of In the Cut is the everyday sound of doors. As prosaic as it seems, doors are an important element

213 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. structuring the movie. In the Cut uses doors to demarcate thresholds, control space, and create zones and spatial containers (discussed below). In particular, Frannie Avery (Meg Ryan), the protagonist of In the Cut, uses doors to create safety by sealing off space. Notable in this regard is the exterior door of Frannie’s apartment building. This door is constructed from 12 separate sonic fragments ranging in type from glass and metal rattles, hinge grinds and squeaks, latch clicks, and wood creaks and thumps. All of these sonic elements are heard every time this door opens and closes but heard in varied orchestration. With every open and close, the intensity of each fragment is mixed in variable relation to the other fragments, and these variations elicit different affective results311. There is a direct relationship between the selection, arrangement and mix of the sonic fragments and the aesthetic response which is their perception. For example, enhancing metal and glass rattles signal a high degree of applied force which, in turn, communicates an intensity of affect whereas a quiet and harmonious balance of sound elements evokes low-intensity affect. A broad range of intensities may be registered in this way, from lethargy to frustration, anger or distress, among other affective tones. This technique wields a double-scoped function whereby affect is sonically induced into the audience and communicates the state of mind of the mediated character within the diegesis to the audience. This is the intercorporal-intersubjective function performed by liberated embodied simulation.

Figures 7.68–7.73. Doors seal off space to create affective acoustic containers. Screenshots from In the Cut (Campion, 2003).

Figure 7.74. A door wipes the story to black in the final moments of In the Cut. Screenshot from In the Cut (Campion, 2003).

Emotion markers, having an extremely limited temporal duration, produce only a “brief burst of emotion” (Smith, GM 2003b, p. 45). Emotion markers, then, exhibit a non-narrative temporality similar to cinema-of-attractions (Gunning 1993, p. 6). This poses a unique problem for analysing cinematic sound. Unlike the temporal continuity of narrative, cinematic sound is fragmentary and associative, and does not yield to the same kind of analysis. Even typical methods for the analysis of music are unsuitable. A sound effect can rapidly arise and dissipate to leave only a residual affective trace, namely, an emotion cue312. Moreover, emotion cues and markers are not necessarily tied to narrative function.

215 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Emotion cues and markers are difficult to analyse using the conventional techniques of self- report and reflection, and do not yield to standard narrative-centric analysis. The analysis of cinematic sound design must proceed in quite a different way. This section responds by cataloguing the tactics of sound design which produce affective moments and contours, a manoeuvre not dissimilar to that used by Philip Brophy in 100 modern soundtracks (Brophy 2004). Brophy’s iconoclastic approach eschews conventional film-theoretical concerns of character and narrative in preference to the listener’s response: those sensations, feelings, moods and emotions aroused by the grain and dynamics of the unfolding sonic events. Brophy’s approach to writing about cinematic sound is a phenomenological one. Indeed, as Brophy sets out, “[e]ach entry is focused on the sensorial, phenomenological and perceptual dynamism of the soundtrack” (Brophy 2004, p. 15), and what arises is a “different mode of writing whose ‘flow’ is more important in its capture, replay and rendering of a film’s momentum, than it is in summarising, reducing or even encapsulating a film’s signifying skeleton” (Brophy 2004, p. 3). However, where Brophy resists formulating a theoretical framework within which to situate his observations I seek it out.

In what follows, I focus upon how sound design inducts affect to generate a network of emotion cues and markers, and provide the basis for Feeling of Body, intercorporeity and intersubjectivity. Encapsulated between 01:00:08 and 01:23:03 (hh:mm:ss) is a major sequence which turns the story of In the Cut as Frannie (Meg Ryan) progresses from sexual fascination with New York homicide detective Malloy (Mark Ruffalo) to discovering the body of her murdered half-sister, Pauline (Jennifer Jason Leigh). In classical narrative terms, the sequence charts the protagonist encountering an apparently insurmountable obstacle which propels her to a high-stakes emotional crisis313.

The sequence commences in a basement laundry where Malloy and a forensic team are retrieving the body of a female murder victim (01:00:08 hh:mm:ss). The scene is visually and aurally cold and harsh. The surfaces are hard, and the colour palette is garishly green, yellow and red. The colours are accentuated by overhead florescent lighting. The sound of water spatters endlessly from the washing machine onto the concrete floor as the forensics team retrieves a limb of the victim. Malloy sits, looking on from the shadows. Malloy’s cellphone rings during the dour, low-energy scene. It’s Frannie. Frannie is at Pauline’s apartment, above a strip club called the Baby Doll Lounge. Malloy excuses himself, exits the laundry, and he and Frannie briefly exchange dialogue. The perspective of their voices is clear, and extremely

216 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. close and intimate. The intimacy of this moment contrasts with the cold horror of the murder scene. During the phone call Malloy exits the building, gets into his police car, and drives downtown. The door of Malloy’s car seals solidly, and the interior becomes sonically annexed from the outside world. Malloy and Frannie engage in phone sex.

The sequence progresses from laundry to passageway to car, from spatially open to closed. The laundry is open, cold and murderous; the passageway is a constricted intermediate space; Malloy’s car is an acoustically compact space enlivened by sex play. Each of these spatial locations is an acoustic container with its own affective tone. The timbre of these locations progresses from cold to warm. Toward the end of their flirtation, a freight truck overtakes Malloy in the adjacent lane. It passes not with vehicular din but a smooth, powerful sensuality. At the scene’s conclusion the music score resolves, weakly, leading to Frannie’s exhale of sexual frustration, and her breath presaging the tonal atmosphere of the incoming subway scene. The music and atmospheres provide a seamless, breathy forward pressure314. Through timbre and spatialisation, the sequence charts a slow tango between emotional distance and intimacy, death and sex, and segues to …

01:03:07 the headlights of a passing car scanning across the wall of Pauline’s apartment, and the heavy sound of its tyres on asphalt. The tyres blur tonally into the sound of an approaching subway train. Taken together, the pattern of sounds signal something large is looming, and that something is revealed as the vision fades through black to a newspaper’s headline announcing murder.

01:03:10 Airy bass tones mark the scene transition to the subway. Frannie stands on the platform, the train arrives, halts. All sound effects are smooth and sleek.

01:03:25 The subway carriage doors slide open with a gentle chime. Frannie looks despondent. She steps into the carriage.

01:03:30 Sonically, the atmosphere of the carriage is subdued but Frannie’s movements are sharp and clear.

01:03:30 Frannie sights her ex-boyfriend, John Graham (Kevin Bacon), in the next carriage but manages to avoid him.

01:03:50 Frannie catches sight of a Poetry in Transit poster in her subway carriage. Frannie recites the text of the poem. Her voice is very close and intimate. As Frannie reads, sound effects of the moving train melt away, and the world dematerialises. Frannie separates from her environment. The sonic texture of the sound effects and atmospheres dovetail with the music so it is difficult to tell where sound design ends and score begins. The Foley of Frannie’s movements is in close perspective and isolated, but her fellow travellers make not a sound. Frannie is in a bubble, mediated by words. As she finishes reciting the poem’s text …

01:04:08 Frannie’s train halts suddenly at the next station, coming to a stop alongside a forlorn bride. Frannie gazes transfixed at the bride.

01:04:14 Compression brakes of the train hiss and perceptually freeze a moment of epiphany for Frannie.

01:04:14 Frannie’s train pulls away from the platform, leaving the Bride stranded. The musical tones and atmosphere dissipate. Cut to:

01:04:24 The door of the police precinct swings closed. Frannie walks into shot. An acoustic guitar plays somewhere offscreen. Malloy meets Frannie and shows her to his desk, revealing Rodriguez playing guitar. The office is busy, noisy. The scene possesses a very narrow visual depth-of-field but the soundfield is continually punctured by offscreen voices and sounds.

01:05:07 Water drips into a bucket. The precinct ceiling leaks from bullet holes.

01:05:40 In the background of the office, a background door squeaks. The sound produces a subtle ‘pip-and-pop’ effect to underscore Frannie’s look of disdain in response to Rodriguez’ question: “I hope you didn’t get the wrong idea?”

01:05:55 Rodriguez produces a file of mugshots for Frannie to look through.

01:07:00 Rodriguez retrieves his guitar, and serenades Frannie. Frannie resists the music, an act of resistance against its entrainment.

01:07:24 Frannie’s attention is deflected by Malloy slamming (offscreen) a filing cabinet drawer.

01:07:31 Rodriguez puts his guitar aside, and asks Frannie if she wants something to eat.

01:07:44 Malloy testily cuts off Rodriguez’ advance, and leads Frannie out of the precinct. Cut to:

01:07:58 Inside Malloy’s police car. The car is an acoustic bubble, but a bubble capable of being shot through with contamination from the outside world in the form of squawk from the police radio. Although the car’s interior is sealed off from the outside world, some sounds do penetrate the membrane of the vehicle, and penetrate in an unrealistic manner. One of these unrealistic sounds is a woman running across the street in front of Malloy’s car. The POV (point-of-view) suggests it belongs to Frannie and, therefore, also Frannie’s POA (point-of- audition). What is unusual about this moment is the woman’s footfalls are clearly heard. Although impossible, Frannie nevertheless hears the running woman. This mirrors an earlier moment in the story where Frannie is first picked up for questioning by Malloy and Rodriguez. From the backseat of the squad car, Frannie observes a young woman frantically running away down a pavement (00:20:13 hh:mm:ss). Sonic linking of this kind, albeit extremely loose, suggests the BRECVEMA mechanism of episodic memory is coming into play.

01:08:02 Another impossible sound heard inside Malloy’s car is the smooth, flowing tones reflected by the bridge pylons as Malloy and Frannie pass to exit the city. (The smoothness of sound contrasts with the flickering staccato pulses heard during Malloy and Frannie’s return to the city. These later sounds are fractured and repetitive, hinting that something is not quite right. The sonic linking of the

219 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. bookends to the out-of-town excursion is subtle but nevertheless exists. The emotional valence turns from positive to negative, from optimistic and promising to anxious and apprehensive.)

01:08:14 Malloy’s car takes an off-ramp, the phasing of its tyres drawing the audience’s gaze to isolate it from the background as the urban gives way to the fringe of the city.

01:08:20 Malloy and Frannie talk candidly as they drive further into the country. –01:09:28 The sequence alternates between interior and exterior visual perspectives but the sequence is unified by a constant interior aural perspective. Their voices are close and intimate. The aural effect produces a bubble of safety, sealed off from the outside world.

01:09:34 The intimacy is punctured when Malloy’s car unexpectedly exits the highway, through a chain marked with a stop sign, and down an unsurfaced road toward woodland. The car breaks the chain with a sharp clatter. Something wrong appears to be happening with the sound perspective of Malloy’s car as it drives away. Aurally, it recedes too quickly. When cutting to the wide shot of Malloy’s car driving away into the woods, the engine and tyre noise recedes unrealistically and, linked to the instability of the visual frame, produces a sense of unease. In essence, this is a form of dematerialisation and reality appears to be dissolving. I used a similar technique in Lantana (Lawrence, R 2001). In Lantana, Valerie (Barbara Hershey) drives to her home on the outskirts of Sydney (approximately 00:48:12 to 00:48:30 hh:mm:ss) and, as her car takes an exit off the expressway, it appears to sonically dematerialise. Although there is a continuity of music score, the car’s engine and wheels on wet asphalt fade. The car drifts away, hushed, as if a ghost car driving on into the night, and a moment of pensive dissociation separates Valerie from audience.

Figure 7.75. Sound design dematerialises the visual image, forewarning danger. Screenshot from Lantana (Lawrence, 2001).

Figure 7.76. Sound design dematerialises the visual image, forewarning danger. Screenshot from In the Cut (Campion, 2003).

01:10:06 Frannie walks with Malloy along a silent lakeside. There is an unsettling lack of birdlife. A gentle rhythm of water laps on the shore. Bags of garbage float in the water. Frannie observes, “This looks like a place where they dump bodies … What’s in these bags?”

221 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 01:10:34 Close-up, water laps gently against a garbage bag. This is an oddly delicate yet powerful moment. What is powerful is the capacity of small and subtle shifts at the micro-temporal level to provide affective beats to provide points of anchor in the affective landscape. The ripples grow quieter, and we are drawn into the visual detail of the shot, examining closely. What is in those bags? The micro- rhythms of the water entrain the listener, drawing us closer. Until:

01:10:38 A gunshot explodes through the bag, erupting garbage. The gunshot echoes and echoes around the lakeshore like a cannon, communicating the lake’s isolation. Cut to:

01:10:45 Malloy approaches Frannie, his pistol drawn. Wind rises up around him, sweeping through the trees315. Frannie retreats, frightened.

01:10:45 Malloy offers Frannie his pistol.

01:11:05 Frannie takes the pistol, unsure what to do with it.

01:11:17 Frannie aims, fires the pistol, almost hitting one of the garbage bags. Echoes ring out around the lake.

01:11:32 Malloy mildly berates her, and Frannie fires the pistol again. This time she hits the garbage bag. More echoes ring out around the lake.

01:11:50 Frannie and Malloy return through the trees to the car. The acoustics of the woodland is dead and anechoic. Frannie, following Malloy’s gesture, sits on the car bonnet.

01:12:20 Frannie unclasps Malloy’s belt. Malloy deflects her advance. Frannie rebuckles his belt. Melancholic music rises. They kiss, but Malloy is distracted.

01:13:41 Malloy breaks away to sit in the car. Frannie reclines half-expectantly. Malloy blows the horn, sharply. Malloy slams closed his door.

222 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 01:13:54 Frannie opens her door, gets in, sees Malloy is upset, and closes the door. The acoustic of the car interior isolates them from the outside.

01:14:01 Malloy asks: “Would you get engaged to me?” Frannie: “Is this is about the murder?” Malloy: “No. Yeah. I can’t get the sick fuck outta my head.” Frannie: “While we were kissing?” Malloy: “All the fucking time.”

01:14:16 Malloy abruptly starts the car engine, and reverses at speed over the dirt. Alternating views of the action cut between interior and exterior, underscoring the acoustic separation of Malloy and Frannie from the outside world. Hard cut to:

01:14:22 Passing under a signal gantry, approaching a bridge at dusk.

01:14:25 NYC, large and grey, flickers between the bridge girders. The sound pulses. The acoustic segmentation of the approach contrasts with the smooth tonal flow of the departure. A speeding, rattling train overrides the flickering city, wiping it to black. At the crescendo, the sound cuts hard to:

Figures 7.77 and 7.78. Returning to New York City at dusk, flickering between bridge girders. Screenshots from In the Cut (Campion, 2003).

01:14:29 Exterior establishing shot of a city eatery at night. The atmosphere is subdued.

223 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 01:14:34 Inside the eatery, Frannie and Malloy talk. The mood is sullen. The atmosphere and sound of the eatery is low-key. Frannie asks if Malloy wants to stop by her place. Things gets complicated. Malloy is going away for a while. Frannie becomes agitated, annoyed. They argue. Frannie gets up and leaves.

01:15:53 Frannie arrives home to her apartment. It is quiet, very quiet. She tosses her keys on the table. Her apartment is in near darkness. Frannie looks absent- mindedly in the refrigerator. Its light spills across the room. She wipes her face, exhausted, closes the refrigerator door, sits, and exhales an uneven, dispirited breath. Then …

01:16:37 Frannie sees a red baseball cap on her kitchen bench. All the while the refrigerator motor grows louder. Until it stops with a judder. Camera focus pulls from the baseball cap to a close-up of Frannie in foreground, barely breathing, turning her gaze slowly, alarmed.

01:16:45 John Graham sits suddenly up into frame. Frannie and John Graham talk, Frannie trying not to be spooked by his erratic behaviour. Hard cut to:

01:17:52 Frannie and John Graham exit Frannie’s apartment building passing through the twinned external doors. The camera remains inside the apartment building as the external door slams closed.

01:18:04 Frannie and John Graham walk along the pavement. It begins to rain. Frannie uses her handbag to shelter from the rain; John Graham does not. As they walk they pass a trio of women sheltering under an umbrella at whom John Graham shouts “Will you have sex with me?!”, and slams his fist into a metal rollerdoor, sounding like a thunderclap.

01:18:19 Frannie descends a stairway to the subway, looking over her shoulder, alarmed, as John Graham shouts. A train rumbles an approach.

01:18:24 Wipe through black to reveal a train pulling away from the platform, the metal bogies shrieking.

01:18:27 Frannie is on the train. The calamity of the rumbling train evolves into airy, industrial and musical tones as Frannie is entranced by more Poetry in Transit. As before, as Frannie recites, the train acoustically dematerialises. A song (Zap Mama’s ‘Allo Allo’ (1999)) rises up, revealing itself to be diegetic music from the incoming scene at the Baby Doll Lounge.

01:18:49 Frannie walks through the crowded Baby Doll Lounge. Women flex on the bar for money. Frannie asks the bartender if she has seen Pauline, but no-one has this evening.

01:19:25 Frannie ascends the stairs from the strip club to Pauline’s apartment.

01:19:34 On the landing, Frannie rubs the belly of a bronze Buddha, and lifts it to retrieve Pauline’s door key. It’s not there. The music continues to play from the club downstairs. Frannie tries Pauline’s front door. It is already open.

01:19:54 Frannie calls Pauline’s name, and hesitantly enters the apartment. Voices and music continue to be heard loudly from the strip club below. Frannie moves further into the apartment. She notices the lamp in Pauline’s bedroom has been knocked over. She goes to it, placing it on a shelf. The action takes place in a medium-wide shot of the apartment.

01:20:29 Close-up, Frannie looks down with repellant wonder at something out of frame. The camera follows her gaze down to a clump of bloodied hair resting on the sheets.

01:20:37 Frannie rises, turning into close-up. Her expression is numb, but expectant. The music from the club below melts away replaced by a growing awareness of water splashing echoically. The shift in acoustic presence is Frannie’s shift in attention and acts to dematerialise portions of the apartment, or – more precisely – materialise an aspect of the apartment which is only now percolating into Frannie’s conscious thought: the running water in the bathroom.

01:20:52 Frannie walks toward the sound, her hands swinging slowly and heavily by her side. Her footsteps are heard, quietly, on the wooden floor. Frannie, silhouetted, moves toward the sound. Cut to:

01:20:52 Frannie in close-up and chiaroscuro lighting, looking steadfastly forward, drawn hypnotically toward the sound of the running water. Cut to:

01:21:17 Medium-long shot. Frannie arrives at a pair of French windows. With both hands, she pulls the doors open. Splashing water and music score surge as the doors open to reveal a steam-shrouded room. Cut to:

01:21:23 Close-up of Frannie as she stares through the steam, down to the floor. A quick intake of breath, face aghast. Cut to:

01:21:24 Frannie’s POV of a blood-spattered bathroom, slowly searching. With a guttural breath, Frannie sees something. A bulging plastic bag in the vanity basin. Cut to:

01:21:40 Close-up of Frannie as she moves slowly, cautiously forward. She opens the bag to reveal Pauline’s dead eyes.

01:21:51 Frannie collapses to the bathroom floor, clutching the head of Pauline. Fade to:

01:22:07 Black and silence. After some time, from the silence, Malloy’s voice whispers Frannie’s name. Cut to:

01:22:16 Slow dripping water. Frannie sits, still collapsed on the bathroom floor, clutching Pauline. Malloy kneels by her side. The room is very, very quiet. Malloy gently tells Frannie he has to take the bag, deftly removes it, and passes it to the waiting forensic team. The bag sounds stomach-turningly heavy. Malloy helps Frannie to her unsteady feet. The room is revealed to be filled with police and the regulars of the Baby Doll Lounge. From the passageway, a distant female wailing rises into the silent room, filling it with its anguish.

226 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. There is no club music nor music score. Frannie walks past sombre faces. Frannie is in deep shock. A man cradles the wailing woman. The energy ebbs from the scene, ending the sequence.

I have heard the end of this sequence many hundreds of times and yet the sound of the inconsolable wailing never fails to cause my breath to choke and tears to well. To me, this sound presents a clear and salient example of the BRECVEMA mechanism of emotional contagion (Juslin et al. 2010, p. 622) where an emotional expression is inducted directly from voice to the listener.

Notice that, of the 20-minute sequence, I have described only the general action and soundscape, limiting any detail relating to narrative development, acting style, cinematography, editing, production design, dialogue or music score. The focus has been constrained to the soundscape of atmospheres and sound effects. Nevertheless, even through such an impoverished description, a number of broad patterns emerge.

The most obvious of these patterns is the slow audiovisual rhythm correlated with a low intensity in affective cuing. The density of cuing is markedly lower than, say, the T-rex sequence of King Kong (examined in Chapter 3). This is not surprising, and fits commonly held assumptions about the genres of arthouse and blockbuster cinema. In In the Cut, only occasional punctuations by loud sound raises the cuing intensity, albeit briefly. These spikes of sound range from the mild effects of doors opening and closing to explosive gunshots. The core affect inducted by the quasi-musical tones of the environmental atmospheres – such as subway rumbles, car and road noise, wind, and the hum of a big city – tends to remain manifest as mood rather than galvanise into emotion. The baseline mood of the sequence is downbeat, tentative and anxious although it evolves from quietly hopeful to utter devastation. It is only when a sound effect, loud and sudden, punctures these atmospheric tones does anything approaching an emotion emerge. Sometimes, a sound only has to be sudden to bring about an emotion. An example of this is the moment surrounding Frannie’s discovery of John Graham’s cap in her apartment (01:16:37 hh:mm:ss). The volume of the apartment’s refrigerator slowly rises and, as the thermostat cycle ends, switches off with a sudden rattle. A beat registers as the camera finds the cap. The leaden silence which unexpectedly follows subliminally cues a scanning of the environment, a sense that something is not quite right.

227 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Another important pattern is the structuring of interior and exterior acoustics. These spaces are psychoacoustically differentiated and, as a consequence, affectively marked316. Although this mode of sonic affect induction is not captured by the BRECVEMA framework it can be explored through the ecological focus of emoacoustics. (Asutay et al. 2012). The binary distinction of inside/outside is used thematically throughout the movie as are a number of other binaries such as up/down, near/far, and city/country (which is itself a re-versioning through story of inside/outside).

Sound not only demarcates space but also describes the motion of objects inside those spaces. Moving objects sound with Doppler shifted frequencies, echo and reverb. These psychoacoustic effects are subtle and rarely rise to our conscious awareness even though we constantly use these capacities to orient ourselves and navigate space. A gentle example of this kind of spatial orientation (better appreciated in 5.1 surround rather than stereo) occurs when Frannie pins Malloy’s business card to her kitchen corkboard (00:16:19 hh:mm:ss). As Frannie lightly caresses the card, a Volkswagen rises in the background and surges across the cut from Frannie seated in mid-shot to a close-up of Malloy’s business card floating in space. The panning and spatial signature of the sound of the Volkswagen approach, the visual edit and accompanying time compression, and the movement of the camera coalesce into a mild disorientation; something approximating the dizzying feeling of infatution.

Figure 7.79. Malloy’s business card swirls in contra-currents of audiovisual space. Screenshot from In the Cut (Campion, 2003).

228 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Auditory looming is an acoustic phenomenon known to induct affect (Bach et al. 2009; Hall, DA & Moore 2003; Tajadura-JiménezVäljamäe, et al. 2010) and enhance orientation to visual stimuli (Leo et al. 2011). Looming refers to a sound which unexpectedly moves toward a listener. It activates the amygdala and triggers alarm (Bach et al. 2008). In the Cut makes use of auditory looming in several scenes. The most important of these, from the perspective of generating an emotion cue which is also narratively significant, is the suddenly approaching wind at the lake (01:10:45 hh:mm:ss). The affect generated by this sound is mapped to Frannie’s fearful face and is rapidly incorporated into the diegesis: Malloy is dangerous. This environmental sound plays out in 5.1 surround acoustic space. A similar looming effect occurs when city ambience, thick, heavy and resonant with subsonics, surges from the rear surrounds to fill the front channels and sweep over Frannie as she emerges from the subway onto a city street (00:11:51 hh:mm:ss). Something ambiguous and powerful is on the move (like King’s “subtext sound”), and this power is directed at the motto emblazoned across a passerby’s t-shirt, or otherwise remains unanchored as mood. Like the lakeside scene, the sound of the city sweeping over Frannie gains psychological power by its movement through a surround soundfield. In contrast to the lakeside scene, the auditory looming in this instance is a form of engulfment or extreme perceptual immersion which kindles a sense of the sublime317.

Figure 7.80. Sound surges over Frannie as she emerges from the subway. Screenshot from In the Cut (Campion, 2003).

229 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Sound design, if it is noticed at all by an audience, tends to be in connection with movies with large or remarkable events, such as in the genres of action-adventure, war, thriller, horror, fantasy or science fiction. It is these kind of spectacle movies which tend to dominate international sound awards and shape the general public’s understanding of what constitutes ‘sound design’. Quiet movies might be equally deftly sound designed but tend to be overlooked come awards season318, and in spite of the invisibility of quietly sounded movies, some have become alert to the spatial sound design deployed in In the Cut319.

Breaking the 20-minute sequence down in the way I have reveals the function of a number of BRECVEMA mechanisms. Firstly, it becomes clear that much of the heavy lifting in the sound design is done by manipulating the brain stem reflex. This may be a function of genre because genre expectations of the erotic thriller hold attraction and repulsion in taut tension, and demand from the protagonist an exceptional alertness of her immediate environment. The brain stem reflex is key because it continually parses the environment, scanning for loud or sudden noises, dissonance, or quickening temporal patterns. Sounds that meet these criteria signal a significant event and cue the reflex in the control of arousal, breathing, heart rate, attention, and movement. Brain stem reflexes rapidly and automatically induct affect, and forge a relationship to the situation without conscious awareness. Such affect is readily projected onto a focalised character. Through sharp sound effects and atmospheric shifts in timbre and spatialisation, the brain stem reflex, more than any other BRECVEMA mechanism, weaves the affective fabric of the sound design. Even so, the sequence also illustrates other BRECVEMA mechanisms, particularly rhythmic entrainment, emotional contagion and musical expectancy.

The induction channels of visual imagery, episodic memory and evaluative conditioning appear to be the most under-employed of all the BRECVEMA mechanisms. Certainly, the mechanism producing visual imagery is difficult to isolate within In the Cut when compared to, say, the Gravity “I've Got You” trailer (Cuarón 2013a). Nevertheless, it can be claimed the image schemata arising metaphorically from sound effects interact in complex ways within the movie’s broader audiovisual imagery to amplify affect or tint it with a particular hue. An example of this is the force of the sexual image of the freight truck which overtakes Malloy after he leaves the crime scene at the basement laundry (01:02:53 hh:mm:ss). The mechanism of episodic memory speaks very squarely to an individual listener’s experience, and activates a broad and largely uncontrollable network of associations which may or may not be

230 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. culturally delineated. Although episodic memory may be a mechanism at play within the sound design of In the Cut it is more readily reflected upon through the music score. Evaluative conditioning is implicated in pairing the vastly expressive array of water sounds with the moral metaphors of contamination and cleansing. The common usage of metaphoric water imagery in artforms other than cinema suggests the potency of this embodied metaphor. Evaluative conditioning may also be argued to form the basis of the emoacoustic relationship between the interior acoustic of closed spaces and safety. Lastly, aesthetic judgement is the most difficult of all the BRECVEMA mechanisms to discuss in specific relation to the sound design of In the Cut, but a few observations can be made in passing. Aesthetic judgement may be construed to be a mechanism felt by an audience who judges the soundtrack to aesthetically satisfy the expectations of a genre. I cannot objectively comment upon the soundtrack’s efficacy and audience response in genre terms because a study of that kind is outside the remit of this thesis; however, it is worth making the observation that issues relating to aesthetic judgement do seem to be a feature in the general reception of the movie. Many movie critics found In the Cut unfulfilling as a genre movie. One need only take a brief look at the range of critics’ reviews aggregated through websites such as Metacritic, Rotten Tomatoes and IMDb. Some are angry, some are disappointed, and others are enthralled.

Water is a recurring motif throughout In the Cut, and it resonates at the familiar level of narrative. It appears as rain, drips, spillages, lake, and river. Water is a common literary metaphor for moral purification but, in the world of In the Cut, water signifies moral pollution and contamination. Summer storms break torrential rain into the New York streets even as an unknown possible assailant surveils Frannie through windscreen wipers struggling to push away the rain (00:33:25 hh:mm:ss). Buckets of water collect in the police precinct as rain leaks through bullet holes in the roof (01:05:07 hh:mm:ss). Lakes are the kind of place where murder victims are dumped, or, at the very least, are polluted by bags of garbage. Water sluices from a washing machine where a murdered woman’s limb has been found. Rain falls upon Frannie as she escapes John Graham’s violent outburst on the street (01:18:04 hh:mm:ss). The rising sound of running water draws Frannie into the bathroom where Pauline has been murdered. Rodriguez attempts to murder Frannie at the lighthouse by the dirty Hudson River (01:45:33 hh:mm:ss). Water is everywhere in the world of In the Cut, and yet water cannot wash away the contamination because water is the contamination. Frannie is immersed in water, and exhausted by water. Concepts drawn from theories of embodied cognition help in understanding how water functions within In the Cut to produce embodied

231 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. meaning. Simone Schnall, discussing the Macbeth effect (think of Lady Macbeth washing her hands), argues moral reasoning relies upon embodied concepts, and observes the emotion of disgust functions as a “guardian of the body's actual and metaphoric boundaries against physical and moral contamination” (Schnall 2014, pp. 233-235). Water in In the Cut is another vector for eroding personal boundaries and safety.

Without doubt In the Cut is a difficult movie. I say this because In the Cut is a collision between multiple registers and modes of address320. It is at once an adaptation of a best- selling novel by Susanna Moore (Moore 1999), an arthouse film by an auteur director, and a vehicle for a Hollywood actress (Meg Ryan) renowned for her roles as the all-American girl- next-door. For Campion it was a bold move, creatively and financially, and was unlikely to satisfy all segments of its potential audience. The movie was very likely to alienate someone, be they fans of the novel who arrive with expectations that Moore’s vision would be faultlessly translated to the screen, those who came to see a Jane Campion film, or those who just wanted to see the next Meg Ryan movie. Genres are labels which signal the types of emotional experience proffered to an audience, and genre expectations vary widely between the broad span of emotion types served up by mainstream cinema versus the typically narrow range of arthouse cinema. Without doubt, In the Cut is an unusual, if not awkward, hybrid: literary adaptation-meets-arthouse-meets-thriller pic. It blends arthouse auteurism with that standard of Hollywood fare the slasher-thriller, and an erotic thriller at that.

In the next section I show how patterns of emotion markers give rise to Feeling of Body, and from Feeling of Body arises Theory of Mind. But first, an elaboration of some conceptual foundations. Mark Johnson and George Lakoff (Johnson, M 1987; Lakoff 1987) have identified a wide variety of image schemata regularly used in conceptual thinking. These schemata may be usefully grouped into categories such as space, spatial motion and transformation, force, and balance. In contemplating which image schema might function as the most fundamental, Schnall (2014, p. 227) identifies several basic properties:

First, the body is a container with a clear boundary that keeps it separate from other objects and people. Second, the body is situated in space and moves in it while maintaining varying distances to objects and people. From these basic properties of the body the following image schemas and associated metaphors are derived: First, verticality provides a source domain to distinguish between good and bad on the most

232 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. fundamental level. Second, the notion that the body is a container shapes the understanding of many emotional and social processes. Third, spatial distance facilitates an understanding of immediate and close, versus distant and remote concerns. Spatial distance relates to objects external to the body that are either kept close, incorporated or instead, are rejected, expelled and condemned. This has relevance for other people, because physical closeness further implies physical warmth, which in itself stands for social connection.

Source domain Area of source domain Target domain

Verticality Valence

Power

Morality

Divinity

Container Depth of Container Emotion

Heat of Fluid in Container Anger

Maintaining boundaries Disgust

Maintaining boundaries Cleanliness

Distance Closeness Intimacy

Warmth Intimacy

Distance Abstraction

Figure 7.81. Basic embodied source domains and corresponding abstract target domains. Adapted from Schnall (2014, p. 230).

In effect, the affordances for action, both physical and social, arise from an affective mapping of space. Fahlenbrach applies similar thinking to affective cinematic space and audiovisual

233 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. metaphor when she notes “mental schemata and the patterns of sensory-motor experience that are the source of most conceptual metaphors are the very basis of filmic space” (Fahlenbrach 2007, p. 118). Fahlenbrach accurately identifies the mode by which multimodal cues mobilise core affect – a mode I call perceptual design. Affect of this kind, as indicated by Russell (2003, 2009), is free-floating and has the capacity to be mapped and targeted onto fictional characters by specific design processes.

To illustrate how an embodied simulation account may be used to analyse the broader meaning-making capacities of cinematic sound, and acknowledging the ideas of Gallese, Wojciehowski, Russell, Fahlenbrach and Schnall, following is a brief analysis of the ‘filmic spaces’ of In the Cut.

7.4.2 Affective acoustics as embodied spatial schema The world of Frannie Avery is organised by a strong spatial design consisting of vertical and horizontal schema. Additionally, there is a third category, but it is less a schema and more a type of slippage: it is a space of leakage, or what Ned Block might call cognitive overflow (see Chapter 6). In sound design terms, these spatial schema are constructed through affectively charged acoustics, emoacoustics. Each of these spatial zones is an acoustic container designed to induct a specific feeling-state in the body of the audience. Recognising the movie to be only sparingly musically scored, a significant proportion of the sonic induction is performed by the sound design of the atmospheres. Once mood is inducted, it acts as an interpretive filter through which the audience perceives events and situations as they unfold within the mediated world.

Vertical schema The vertical schema of the movie partitions space into four zones: i. High Above Street-Level – a highly protected zone. In this zone the world is kept forcibly at bay. This is the level where Frannie’s apartment is located, and it is the movie’s most secured and hushed space. It has a high degree of order and control imposed upon it. There are no sharp or transient sounds in this space, no surprises. ii. Above Street-Level – a zone only partially protected, and the world cannot be kept at bay. This is the level where the apartment of Frannie’s half-sister, Pauline, is located, above a

234 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. strip-club called the Baby Doll Lounge. Pauline resorts to sleeping pills to deaden herself to the invasion of noise seeping up through the floor from the Lounge below. iii. Street Level – a zone of intrusion and confusion, with no clear demarcation of spatial limits. This is the level of streets and cafés, of the Red Turtle bar and the Baby Doll Lounge. At this level, psycho-physical boundaries break down, and the ability to control space and personal security is weakened. Sound seems chaotic, and liable to jump suddenly and unpredictably into awareness. iv. Below Ground – a zone which blends fear, sexuality, desire and fantasy. It is a morally ambiguous space. This is the level of the Red Turtle’s basement and the subway. In this zone, acoustic and psychological space distorts, and the differentiation between inside and outside blurs. What is psychologically interior and physically exterior is unclear or unformed.

Each of the above demarcate both physical and emotional space by shaping a specific sonic signature. Such vertical spatial schemata also act as symbolic organisation. However, while the vertical organisation of space is palpable, a horizontal spatial schema also operates.

Horizontal schema The horizontal schema of the movie partitions space by proximity. Close proximity affords not only intimacy and sexual encounter but also violence and death, while remoteness affords safety and security but also loneliness. The horizontal schema is controlled by a binary interior/exterior logic, and is expressed through three zones: i. Super interior – a thought space, such as Frannie’s interior monologues on the subway. A good illustration of this kind of super interior space occurs on a subway carriage at 00:11:18 (hh:mm:ss) where Frannie recites the poetry of Lorca but her voice and lips are desynchronised, suggesting her voice issues from a different space to the reality in which her material body exists. ii. Interior/Exterior – space segmented by personal bubbles of proximity321. Proximity is gauged as radiating out from Frannie’s own body and thus maps motor space. One of the functions of point-of-audition is to map such egocentric spatial radiation. Another example of the interior/exterior schema is the conspicuous and extended use of doors to seal/unseal

235 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. spatial containers, such as rooms and cars, and the psychological import of characters moving across thresholds. Conventionally, characters moving through doorways are edited from the scenic montage to speed up dramatic action. In the case of In the Cut, the sealing/unsealing of secure space is essential to the dramatic action and is enhanced. iii. Near/Far – controlled by movement, this is a variation of the above kind of space segmented by proximity. A good illustration of near/far space occurs at the Red Turtle bar when Frannie meets with one of her students. Although the conversation is amiable, Frannie remains characteristically reserved if not a little on edge. Suddenly, a glass smashes, and Frannie’s attention jags to the culprits, a rowdy group playing pool at the back of the bar (00:06:40 hh:mm:ss). The glass smash, although it occurs at some distance, acoustically punctures Frannie’s personal space and shifts her mood toward irritation and impatience. Near/far space is also dynamic in that people or objects may be categorised as either moving toward Frannie or away. This type of schema is, in part, enabled by the psychoacoustic effect of auditory looming (Hall, DA & Moore 2003; Neuhoff 2001), and has been discussed above in relation to the rising wind at the lake as a sonic event (01:10:45 hh:mm:ss).

7.4.3 Affective zones and cognitive overflow As I have already indicated, there is also a third schema at work, but it is less a schema than a form of leakage or ‘non-containment’. Examples of this include Frannie becoming overwhelmed by the sonic pressure of the city soundscape as she emerges from the subway (00:11:51 hh:mm:ss); the sonic freezing of time as Frannie senses an intruder in her apartment (01:16:34 hh:mm:ss); the aural elasticity and dematerialisation of Pauline’s apartment as Frannie enters the steam-shrouded bathroom to discover her half-sister’s murder (01:20:46 hh:mm:ss). I consider these all to be moments of cognitive overflow: when the pipeline of narrative consciousness cannot immediately nor cohesively contain the minutiae of phenomenal information322. A potent characteristic of this kind of affectively driven overflow is the cinematic manipulation of subjective time, particularly techniques which emulate the phenomenon of time slowing when confronted by danger323.

7.4.4 Feeling of Body and the construction of character The primary function of sound design in In the Cut is simulation of the experience of persistent yet non-specific threat, of anxiety and exhaustion, within the body of the audience. Once simulated as Feeling of Body (FoB), the cognitive consequences of these feelings may

236 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. be projected onto a fictional character. These processes of liberated embodied simulation provide the audience with a mechanism by which they may ‘know the mind’ of a fictional character by forming a folk-psychology based upon affective tone and perceptual realism. This explanation resonates with Fahlenbrach’s observation that

[f]ilmic spaces are rarely recognized consciously by the viewers. At the same time, their perception and interpretation of the narrative is highly guided by their bodily experience of the filmic space. (Fahlenbrach 2007, p. 105)

The global effect of In the Cut’s sound design is to construct Frannie Avery as a character who lives in a state of siege. Frannie walks through a post-9/11 New York, padded and cocooned inside a psychological bubble. Within the movie’s soundscape there is no mid- ground, only a close and intimate foreground or a remote background. Because the sonic middle-space is missing, Frannie’s personal zone is vulnerable to unexpected puncture, triggering alarm. This sonic dampening creates an aural version of the ‘thousand-yard stare’ suffered by traumatised combat soldiers. In the case of In the Cut, the connotations of battle trauma are mapped in terms of gender politics. Moreover, the psychological bubble within which Frannie lives is highly mediated by language324; it acts as both a conduit to the sensory, emotional and social, and a door which opens or closes to control space and limit risk.

The folk-psychological Theory of Mind by which we come to understand Frannie Avery’s character, desires and motivation emerges from the Feeling of Body and proto-narrative generated by the perceptual design of multimodal cinematic cues. The induction of Feeling of Body is a fundamental cinematic technique and it manifests in weak or strong form within every mediawork regardless of genre. Design strategies similar to those used in In The Cut are highly pronounced in Paul Thomas Anderson’s Punch-Drunk Love (Anderson, PT 2002) and Alejandro Iñárritu’s Babel (Inárritu 2006). For example, in the opening sequence of Punch- Drunk Love the insistent mismatch between point-of-audition and point-of-view is mapped onto the central character of Barry Egan (Adam Sandler) so that the evoked cognitive dissonance comes to be understood as deep anxiety and overwhelming social paralysis325. In the city nightclub of the ‘Tokyo Night’ sequence of Babel (01:02:29–01:12:20 hh:mm:ss), the deaf protagonist Chieko (Rinko Kikuchi) rides ecstatic waves of party drugs and meshing and unmeshing musical rhythms which climax with an EDM drop, a drop which announces a harsh turn in the narrative326. The purpose of each of these design strategies is to induct a

237 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. feeling state within the body of the audience and come to understand the thoughts of a focalised character through the processes of liberated embodied simulation.

7.5 Summary: a biological basis for cinematic feeling and thinking In summary, this chapter applies the findings from Chapter 3 in the function of affect within processes of multimodal integration and mental imagery formation, from Chapter 4 in the primacy of affect and its function within theories of embodied cognition, from Chapter 5 in the role of auditory induced affect in an ES account of cinematic media, and from Chapter 6 in the function of emotion in generating consciousness and narrative. Specifically, the chapter examines the aesthetics arising from the embodied basis of cinematic deep structure.

The central claim of my embodied simulation account proposes the primary function of cinematic sound to be the production of meaning through the induction and structuring of affect within the body of the audience. Fundamental to the account is the auditory induction of affect, and the elicitation of mood and emotion through exposure to audiovisual imagery. To recap, the mood-cue approach recruits and scaffolds upon the auditory induction mechanisms described by the BRECVEMA framework, among others. The mood-cue approach builds upon the BRECVEMA mechanisms by adding the polysemous dimensions of crossmodal interaction. Like the BRECVEMA framework, the mood-cue approach describes how nonconscious affect may be mobilised through the perceptual design of cinematic imagery. Once mobilised, this affective content becomes available for further processing. Emotional communication theory describes how this subsequent processing scaffolds from the low-level neurobiology of affect to the tertiary levels of cognitive appraisal, the structuring of emotion prototypes, and the social construction of emotional experience.

Cinematic media routinely engineers a form of synaesthetic experience through the technological recombination of aural and visual imagery. When manipulated, the body’s innate processes for multimodal integration produce novel audiovisual metaphors which, in turn, guide and enhance processes of cognitive blending in the production of thought. Moreover, affective bursts of sensation, when organised into patterns, form the basis of proto- narrative. Proto-narrative acts as a template upon which narrative proper is based, and strategies of sound design assist an audience in coming to know a fictional character by simulating a Feeling of Body which then affords the extrapolation of a Theory of Mind.

As this chapter has demonstrated, embodied simulation theory is strategically placed to explain the relationship between the human body, consciousness and cinematic media because it places paramount importance upon the transduction of sensorimotor information into embodied meaning. Embodied simulation in its ‘liberated’ form evoked through artworks places the nonconscious mechanisms of the body at the centre of human understanding. Thus, cinematic media, as a homologue of human feeling and thinking, operationalises embodied simulation as a cognitive tool which permits the technological extension of the human mind327.

The primary function of the sound effects and atmospheres of In the Cut is the induction of anxiety and desire with occasional bursts of fear, panic, horror and triumph. A slowly modulating audiovisual rhythm unfolds throughout the movie to induct core affect and mood and, through the shifting intensities of the sound effects, coalesce into emotion markers. These emotion cues and markers produce an affective network which allows the audience to associate and resonate screen events with an individual’s memories and broader life experience and thus create wider interpretations of the movie. Patterns of core affect are focalised upon Frannie through standard techniques for producing point-of-audition and point-of-view, and the outcome for the audience is a cinematic Feeling of Body. Feeling of Body is enabled by a liberated form of embodied simulation (Wojciehowski & Gallese 2011, p. 2). During the practice of sound design, intercorporeity is inducted and upon this intersubjectivity is built.

------this page intentionally blank ------

We shall not cease from exploration The end of all our exploring Will be to arrive where we started And know the place for the first time. – ‘Little Gidding’, T.S. Eliot (1942)

8.1 Embodied cognition and the primacy of affect

With the conclusion of this research, I am returned to where I began: I understand the power of cinematic imagery lies in the capacity to experience the world as if from within someone else’s skin, and I have come to understand why and how this is achieved. At the end of the doctoral research process, I know what I have always tacitly known but in a way that is less tacit and more utterable and communicable. In this concluding chapter I take stock of the main points of my argument, and make explicit the fresh understanding that has been generated and delivered throughout the previous chapters.

8.2 Research goals

Claims, hypotheses, and methodology Cinematic sound is philosophically straitjacketed and disenfranchised by standard theories within Film Studies as a consequence of an overdetermining ocularcentrism and narrativism, and a mind–body dualism which haunts hermeneutic interpretation. A novel account is required to more comprehensively explain the role and function of sound design within cinematic media.

Classical approaches to film-sound are lacking because they privilege the study of speech and music while failing to account for environmental sound. Similarly, contemporary cognitive

241 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. media theory is largely silent on the sense-making functions of general sound design. Speech and music are both socially produced and encoded forms of sound which may be readily but solipsistically explained using conventional cultural theory. However, speech and music comprise only a fraction of a vast sounding universe, and what is not accounted for in any sophisticated fashion is environmental sound. Recognition of this omission encourages the pursuit of a very different mode of inquiry, one which privileges evolutionary and ecological approaches to audition.

My research commenced from the hypothesis that the design of cinematic sound is comprised of perceptual and narrative factors which interact to produce a meaning-making system. Specifically, it was proposed that the perceptual design of cinematic imagery drives bottom- up nonconscious affective processing in its audience to produce conscious thought and narrative flow. This represents an inversion of the top-down classical account.

My research methodology explored the hypothesis by cross-mapping the techniques of cinematic sound design I have acquired through professional practice against contemporary research in the cognitive sciences. The tightest fit and most productive resonance of this cross-mapping occurred within the research paradigm of embodied cognition, particularly within theories of embodied simulation. Embodied cognition rejects mind–body dualism; it dissipates ocularcentrism by conceiving of the human perceptual system as a system; and it contextualises narrativism within a much larger system of meaning-making processes. In short, embodied cognition neutralises precisely those philosophical biases which have disenfranchised cinematic sound for a century.

Aims and objectives The primary aim of the thesis was to abstract key concepts from sound design praxis and match these against current concepts in embodied cognition and consciousness studies research. The objective was to develop a conceptual framework for understanding the nature and function of cinematic sound. The methodological objective was to gain a basic scientific understanding of the perceptual, affective and cognitive processes which underpin the creation, consumption and effects of cinematic imagery and, in so doing, forge a consilience between this form of understanding (causal explanation, or Erklärung) and the form of understanding typically deployed within the humanities (interpretive understanding, Verstehen)328.

242 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. 8.3 Research outcomes The major outcome of this research is the origination of an embodied simulation account of cinematic sound design. Borne from the interrogation of a series of professional intuitions, the research investigated regularities within sound design praxis and these regularities found conceptual resonance within cognitive science, particularly studies in embodied cognition. The EC paradigm proposes cognition to emerge from sensorimotor processing, and these processes provide the foundation upon which consciousness, abstract thought and imagination are scaffolded. Imaginative acts, in particular, re-purpose image schema to simulate fictional or counter-factual scenarios in an offline fashion329.

Central to the embodied simulation account of cinematic sound is its function in generating meaning through the affective experience of sensorimotor cues, and this insight extends to cinematic media more generally. Film Studies has conventionally focussed upon vision and narrative as the engine for meaning-making, and this determining ocularcentrism and narrativism has granted comparatively little significance to the soundtrack. In contrast, an embodied cognition approach provides an innovative strategy to assay these conventional assumptions and, in contrast, construes cinematic media to be powered by the nonconscious mechanisms of liberated embodied simulation.

Accruing empirical evidence reveals sense modalities do not function in isolation, and encourages a view of the human being as a thoroughly multimodal creature. In turn, this encourages both a scepticism of the maxim that cinema is a purely visual medium and a recognition of the integral role audition performs in the cinematic experience. Because aural, visual and motoric information is automatically integrated by the MNS, sound design is inherently involved in the process of simulation. Furthermore, arguments from embodied cognition theory indicate these processes operate before and below the threshold of conscious thought and the self, and that affect plays a role, firstly, in binding multimodal information into percepts (Collignon et al. 2008; Pourtois et al. 2000) and, secondly, providing a background of core affect that lends valence to the environment and shapes cognition as a function of embedded action (Russell 2009; Winkielman, Niedenthal & Oberman 2009). This reflects Zajonc’s argument for the primacy of affect (Zajonc 1984), and suggests itself as an active force within Mark Johnson’s concept of embodied meaning (Johnson, M 1997, 2007b).

243 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Embodied simulation (Gallese 2005) is a specific theory within the paradigm of embodied cognition. Embodied simulation theory describes a universal, automatic and nonconscious mechanism which “mediates our capacity to share the meaning of actions, intentions, feelings, and emotions with others, thus grounding our identification with and connectedness to others” (Gallese 2009, p. 524). In the context of literature and the arts, intercorporal resonance underpins what Wojciehowski & Gallese (2011) call Feeling of Body: a ‘liberated’ form of embodied simulation of what it feels like to be a body other than your own. Embodied simulation theory uses this mirroring function to explain the emergence of social cognition, Theory of Mind, and empathy in human beings.

Embodied simulation theory provides an organising framework for this thesis wherein other specific and highly granular theories for the auditory induction of affect and its cinematic structuring may be nested. In particular, I recruited the BRECVEMA framework (Juslin 2013b; Juslin et al. 2010; Juslin & Västfjäll 2008) to explain how aural imagery inducts affect; the mood-cue approach (Smith, GM 1999, 2003a) to show how cinematic aural and visual imagery stylistically generates an ecology of embodied meaning which is independent of and yet vital to the emergence of narrative; and emotional communication theory (Bartsch 2004, 2005; Bartsch & Hübner 2005) to integrate neuropsychological and social constructivist theories of emotion in recognition that a central function of communication is the mutual sharing and influencing of emotions. Taken together, an embodied simulation approach provides cinematic sound with a powerful explanatory architecture for the induction and structuring of affective content by scaffolding the BRECVEMA framework, the mood-cue approach, and emotional communication theory.

The thesis also developed the proposal that embodied and narrative consciousness have their cinematic homologues in perceptual and narrative immersion, and proto-narrative acts as the interface between these forms of design. To illustrate how cinematic sound creates embodied meaning, this thesis examined sound design strategies used in a range of music video, TVCs and blockbuster movies, as well as Jane Campion’s arthouse movie In the Cut. In the context of a character-driven narrative, such as In the Cut, it was shown how the strategies of sound design assist an audience in coming to know a fictional character by simulating a Feeling of Body which then affords the extrapolation of a Theory of Mind. In this way, sound design creates embodied meaning through the interaction of perceptual and narrative immersion.

8.4 Research conclusions The above research outcomes support my professional intuitions concerning the multimodal and affective primacy of cinematic media, and allow a number of conclusions. Firstly, audition and vision are complexly entangled. Audition modifies visual perception, and vice versa. Sound design has the capacity to shape and steer visual attention, and may interact with vision to produce synaesthetic experience. The complexity of dynamic crossmodal interaction leads to the conclusion that cinematic imagery is not simply located in the visual but is profoundly multimodal. Secondly, what is cinematic about cinematic media is not moving pictures per se but the affective content immanent to moving imagery. To this end, the primary function of sound design is the generation of affective moving imagery and, as a consequence, embodied meaning. Thirdly, because proto-narrative may supply a satisfying experiential episode in its own right, it may be concluded that the primacy of cinema is not strictly narrative but must be affective. Proto-narrative provides the foundation for narrative proper, but it is neither necessary nor required that narrative be ultimately generated.

Converging evidence within the cognitive sciences provide sophisticated models as to how neurobiological mechanisms produce embodied meaning. These models assist in explaining why sound design techniques and technologies have developed in the ways that they have, and how they leverage embodied mechanisms for sense- and meaning-making.

8.5 Limitations and caveats to research In summarising this thesis’ argument for an embodied simulation account of cinematic sound, several research limitations and caveats must be acknowledged. Firstly, theories for the auditory induction of affect remain under-developed; secondly, language-centric analysis presents specific problems when theorising cinematic media; and thirdly, sound is yet to fully slip from its philosophical straitjacketing.

Theories of auditory induction of affect remain under-developed The first caveat concerns the BRECVEMA framework itself. In drawing upon this framework I recognise that limitations arise from the model’s explanatory focus upon the musical induction of affect. Musical listening, as I proposed in Chapter 5, accounts for only a fraction of the possibilities of a sounding universe330. The perceptual rules of music cognition

245 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. developed from an earlier and more fundamental form of auditory cognition and it is possible, even probable, the BRECVEMA framework does not capture all extant mechanisms. Juslin and Västfjäll acknowledge this shortcoming when they write “music evokes emotions through mechanisms that are not unique to music” (Juslin & Västfjäll 2008, p. 559), and note many of the BRECVEMA mechanisms “involve low-level processes that developed before music even existed, and that consequently don’t treat ‘music’ as a distinct type of event” (Juslin 2013b, p. 241). These comments suggest the current iteration of the BRECVEMA framework may omit mechanisms which are not necessarily musical but ecological in nature – mechanisms keyed into Gaver’s notion of an everyday listening (Gaver 1993a, 1993b) – but this is merely a sin of omission not of approach. The BRECVEMA framework is a work-in-progress, as Juslin’s own revisions of the model illustrate. There is much work yet to be done to exhaust the research possibilities of ecological listening and the auditory induction of affect. One area rich for future research, for example, is the auditory and affective functions of the brainstem, the most phylogenetically ancient part of the brain and the seat of consciousness.

Cinematic media and the problem of language-centric analysis The linguistic distortions and biases within Film Studies impose several types of limitation upon any deep analysis of cinematic imagery, and, as Chapter 2 has shown, the consequences of poor terminology is poor theory formation. “[T]extual analysis”, Sergi states baldly, “has been remarkably impervious to all things sound" (2004, p. 136). Additionally, the many, many cross-disciplinary terminological confusions and misappropriations I have encountered while researching this thesis present unique linguistic obstacles in their own right, and these obstacles act against a frictionless interdisciplinarity. More serious, however, is a related problem hovering in the background of this thesis: the bittersweet irony in attempting to explain the embodied meaning of cinematic sound through the written word. This is the problem of ineffability. Ineffability is the “degree to which percepts or concepts resist linguistic coding” (Levinson & Majid 2014, p. 407), and is traceable not only to the limits of language but also of human cognitive architecture. Ineffability presents a sobering obstacle for this thesis (if not Film Studies generally) because the object of research is the design and registration of qualia, those “subjective, phenomenal aspects of experience which seem to be essential for individuating sensations and feelings” (Levinson & Majid 2014, p. 408) and which are, of course, the very nature of cinematic imagery.

246 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. The nature of embodied meaning, as this thesis professes, is affective and pre-verbal. Embodied meaning is unavailable to direct scrutiny by conscious thought. Written words fail to capture the entirety of such meaning because precision and vitality is lost from the communicated affective image. Language, as Johnson (2007b, pp. 88-89) reminds us, slices across perceptual flow to produce those static quasi-objects known as concepts. One of the consequences of written language, therefore, is abstraction and reduction. Language scaffolds upon – but abstracts away from – the embodied basis of the sensorimotor system while remaining parasitic upon it (Cuccio 2014; Gallese & Cuccio 2015). Embodied meaning cannot be understood through words alone, and for this reason excerpts from the mediaworks under analysis are available on the DVD-ROM which accompanies Appendix D to exemplify the featured cinematic mechanisms. These media files, where possible, preserve the original soundfield information (e.g. mono, stereo, or surround) at the highest available resolution. Additionally, screenshots have been taken directly from the mediawork and provided as keyframes in the body of the text to assist reader orientation.

Slipping sound theory from its philosophical straitjackets In Chapter 1 a series of powerful philosophical biases were identified which straitjacket the theorising of cinematic sound. The first is ocularcentrism, challenged in Chapter 3, which assumes vision exercises hegemony over audition. The second is cognitive primacy, challenged in Chapter 4, which assumes cognition commands dominance over emotion. The third is interpretive meaning, challenged in Chapter 5, which is assumed to be the prime exemplar of human signification and is the province of language and culture. The fourth straitjacket is narrativism, challenged in Chapter 6, which assumes conscious thought and narrative commands and controls the body’s sensorimotor system and its associated imagery331. All four biases are endemic within Film Studies and act historically to form an interlocking hegemony which subordinates and disenfranchises cinematic sound and the unique forms of meaning it propagates.

While this thesis cannot deliver fatal blows to ocularcentrism, cognitive primacy, interpretive meaning or narrativism, it does cast doubt upon the authority of these precepts in determining a theory of cinematic media.. Behind these precepts, and beyond the scope of this research, is the spectre of mind–body dualism which haunts the interpretive mode of analysis so dominant within contemporary humanities discourse. These routines of thinking are too deeply entrenched in Western philosophy to be undone here. My aim, however, is to provide limited

247 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. examples from a broad range of evidence as to why the authority of these conventional arguments should be doubted so as to loosen the straitjackets and allow cinematic sound to breathe a little fresh air.

8.6 Research contributions and implications The research outcomes of this thesis contribute to contemporary Film Studies in a number of ways. Firstly, a key contribution is the development of a conceptual framework which explains not only the purpose and function of sound design but also cinematic media more generally. The hermeneutics of standard film theory have failed to comprehensively explain the function of cinematic sound, and an embodied simulation account proposes a more flexible and inclusive model which does not necessarily negate existing cultural levels of explanation but grounds them at a profoundly deep biological level. Central to the account is the function of embodied affect in the processes which give rise to meaning. Taking this insight as foundational, my account scaffolds the BRECVEMA framework, the mood-cue approach, and emotional communication theory to provide a model of cinematic media which is both tightly integrated and highly scalable from the sub-personal to the personal to the cultural. This account contributes new knowledge, and extends current understanding by offering a more sophisticated model than its classical predecessors which tend to offer explanation at only a cultural level.

Secondly, my proposed notions of perceptual design and proto-narrative (which emphasise affective imagery and immersion) operationalise embodied meaning. A corollary of these concepts is the elevation of cinematic spectacle to a foundational form of sense-making worthy of study in its own right. Such a stance reverses the conventional evaluation of spectacle as something in excess of narrative knowledge and therefore worthless if not ideologically suspect.

Thirdly, an embodied simulation account successfully places cinematic media within an evolutionary framework, and thus responds to the call for an ecological metatheory (Anderson, JD & Anderson 1996). Such an evolutionary framework presents cinematic media as a cognitive tool and a rich biocultural adaptation332.

248 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Lastly, the exceptionally good fit between professional media praxis and embodied simulation theory suggests the implicit knowledge possessed by mediamakers renders them “practical cognitive psychologists” (Bordwell 2008, p. 143), and suggests a continued study of the ways mediamakers operationalise implicit theory could yield yet more important insights.

8.7 Directions for future research and possible application of research findings The conclusions I have drawn from an embodied simulation account of cinematic sound and media suggest a number of important future avenues for research and application: firstly, the prevailing importance of continued monitoring of developments in cognitive science, particularly with regard to the BRECVEMA framework and theories of embodied cognition more generally; secondly, a deeper examination of the underlying relationship between cinematic spectacle and narrative; thirdly, an appraisal of cinematic media as an investigative probe into the dynamics of the body–mind; fourthly, extended investigations by which to bridge the apparent strangeness of an embodied simulation account of moving image media with standard Cultural Studies; and, lastly, the implications of an embodied simulation account of cinematic media in relation to individuals and institutions involved in cultural production and media education.

Monitoring of developments in cognitive science The shortcomings of the BRECVEMA framework, outlined above, suggest an immediate line of enquiry. While legitimate from an evolutionary perspective, the BRECVEMA framework requires further elaboration to produce a truly exhaustive explanation of the embodied meanings generated by everyday listening and environmental sound. The limitations of the BRECVEMA framework suggest future research should be directed toward the neurobiological and neuropsychological functions of the auditory brainstem and thalamus333. Additionally, theories of embodied cognition and embodied simulation have major repercussions for theories of aesthetics, art and entertainment, and the promise of such a re- evaluation presents itself in the recently emerged field of aesthetic science, also known as neuroaesthetics334.

Rethinking the spectacle / narrative distinction A counterintuitive outcome of this thesis is a scepticism toward the utility and veracity of the spectacle/narrative distinction within standard Film Studies. Contemporary neurobiological

249 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. theories of consciousness provide sophisticated models whereby the mind–body dualism which underlies the putative spectacle/narrative distinction may be expunged, and so open the way to a broader understanding of the nature of interaction between these two structures of meaning335.

Investigating the body–mind nexus through cinematic media This thesis has shown a tight relationship exists between cinematic media, affective nonconscious processes, and conscious thought. This uniquely positions cinematic media in exploring the interplay between body and mind336. For example, in Chapter 7, proto-narrative was proposed as a conceptual amalgam of embodied mechanisms which provides affective shape to cinematic experience. Moreover, it was proposed the affective shape of proto- narrative underpins the emergence and function of narrative proper. Because I functionally equate cinematic proto-narrative with Damásio’s core consciousness (Damásio, AR 1998), proto-narrative could be developed as a useful experimental mechanism by which to probe337 Baumeister & Masicampo’s proposition of consciousness as an animal–culture interface (Baumeister, Roy F. & Masicampo 2010) and the Unconscious Thought Theory developed by Dijksterhuis & Nordgren (Dijksterhuis & Nordgren 2006) and Bargh (Bargh 2011).

Bridging cognitive science and Cultural Studies Within Film Studies a discussion of embodiment and cinematic affect would ordinarily invoke the work of Vivian Sobchack (1992, 1994, 2004) and Laura U. Marks (2000), as well as the ‘cine-philosophy’ of Gilles Deleuze338 (1986, 1989). However, I have not drawn directly upon these thinkers because my conceptual approach is via the perceptual and affective sciences. In some ways, my embodied simulation approach can be said to be ‘before and below’ that of Deleuze. The horizon of my research traces to where the cognitive sciences interconnect with this film-theoretical approach, and it is possible an embodied simulation account of cinematic media offers a testable neurobiological basis to Deleuzian film theory339. Conversely, it is possible Deleuze’ work offers theories of embodied cognition a bridge to cultural theories of subjectivity, gender, race, society, and culture.

Cultural production and media education The conceptual framework of visual narrativism not only frames the central assumptions of Film Studies but organises broader governmental, industrial and educational policies concerning cultural production and media education. An embodied simulation approach, as

250 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. this thesis has shown, provides a thoroughly different account of the forces at work in the production and consumption of cinematic art and entertainment. Indeed, an ES approach prompts for the development of a very different set of policies governing intellectual property and the creative industries; it suggests the need to reconfigure project development and recalibrate the allocation of creative resources. The implementation of a conceptual framework drawn from theories of embodied cognition, therefore, has the potential to radically restructure the traditional relationship between what is known in the industrial context as above- and below-the-line personnel340. Moreover, such a conceptual framework welcomes and understands emerging media technologies (such as augmented and virtual multimodal environments) and prepares the way for their speedy integration into the industrial and cultural media landscape. In the context of media education, the adoption of an embodied cognition approach means shifting away from the traditions of visual narrativism toward a broader and richer paradigm which offers the potential to bust and radically reorganise existing silos of knowledge. Ultimately, a more nuanced and accurate understanding of the forces which underlie the production and consumption of moving image media wields the potential to better inform decisions made by individuals and institutions involved in future policy development, cultural production and media education.

8.8 Concluding remarks The primary function of sound design is the creation of core affect through the perceptual design of cinematic imagery. In the cinematic, we experience a mediated world simulated as if through another body, a Feeling of Body. This Feeling of Body is facilitated by a technological production of presence which exploits the tight fit with which our sensory and perceptual systems evolved to interface with the real world. During the practice of cinematic sound design, what is designed is a prescribed experience of affect, moods, feelings and emotions and, hence, thoughts, for “emotions configure our cognitive systems” (Oatley 2009). Cinematic sound design, cognisant of Johnson’s argument for a bodily based theory of aesthetics (2007a), is an embodied process of experiential knowing.

Finally, on the historically intractable problem of dualistic thinking which cleaves body from spirit, Sontag’s famous declaration of the need for an erotics of art appears as astute and relevant as when she published in 1966. Sontag’s urgency to “recover our senses” impresses upon me the importance of how embodied cognition and a science of aesthetics, a

251 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. neuroaesthetics, might usefully “show how it [art] is what it is, even that it is what it is”. There is much work to do, and we should act more like embodied creatures and less like frustrated angels. The body is not a tabula rasa but the font of meaning.

* * *

Below is a précis of the emotional communication theory developed Bartsch & Hübner (Bartsch 2004, pp. 1-3): i) Neuroscience models take emotions to arise as a result of the activity of brain systems specialised in two functions. The first is the analysis of sensory information for “emotional meaning” which comes about via “associative pairing of stimulus features with innate emotion elicitors” (Bartsch & Hübner 2005, p. 2). The second is controlling the autonomic nervous system and hormone release, as well as attention and other cognitive resources. These two functions reveal the mechanisms by which the body’s emotion system automatically analyses the environment and prepares it for action based upon noncognitive ‘emotional meaning’. This theoretical approach is at the level of neurobiology hence the apposite term of its study as affective neuroscience. Examples of affective neuroscience modelling may be found in the work of António Damásio (1994), Joseph LeDoux (1996), and Jaak Panksepp (1998b). ii) Appraisal theories consider emotion to arise from cognitive appraisals of such criteria as “novelty, valence, goal conduciveness, certainty versus uncertainty of outcomes, self versus other’s agency, coping potential, and normative evaluation” (Bartsch & Hübner 2005, p. 2). On the view of appraisal theorists “[e]ach emotion is caused by a specific pattern of appraisals. Emotion expression is, therefore, assumed to carry detailed information about cognitive appraisals” (Bartsch & Hübner 2005, p. 2). Examples of appraisal theory may be found in the work of Richard Lazarus (1991), Klaus Scherer (1984b), Nico Frijda (1986), and Andrew Ortony and Gerald Clore (1988). iii) Prototype approaches assume “knowledge about emotions is represented in the form of nonverbal emotion scripts”, and these scripts “include knowledge about typical eliciting situations, typical reactions, and self control procedures” (Bartsch & Hübner 2005, p. 3). Emotion prototypes act to structure a personal experience of emotions and the recognition of emotions in others. The work of Phillip Shaver and colleagues (Fischer, KW, Shaver & Carnochan 1990; Shaver et al. 1987) provide examples of the prototype approach.

253 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. iv) Social constructionist theories assume emotional knowledge to be represented in a symbolic manner and the “rules that govern the use of emotion words and other symbolic expressions are thought to be equivalent to the social norms concerning emotions”; that is, “emotions can be considered as social roles” (Bartsch & Hübner 2005, p. 3). Social constructionist theories are illustrated by the work of James Averill (1985), Rom Harré (1986), Richard Shweder (1994), and Catherine Lutz (1988).

Below is a table of hypotheses for the original seven BRECVEM mechanisms for the musical induction of emotion. Note that this model does not include the final mechanism of aesthetic judgement. Adapted from Juslin (2013b, pp. 243–244):

Mechanism Phylogenetic Survival value Information Mental Order of brain focus representation function

Rhythmic 2 Facilitating Periodic pulses in (Cued) entrainment motor rhythms, sensorimotor coordination in especially around representations physical work 2Hz tasks

Evaluative 3 Being able to Co-variation (Cued) conditioning associate objects between events associative or events with representations positive and negative outcomes

Contagion 4 Enhancing group Emotional motor (Cued) cohesion and expression sensorimotor social reminiscent of representations interaction, e.g. human voices between mother and infant

Visual imagery 5 Permitting Self-conjured (Detached) internal visual images pictorial simulations of representations events that substitute for overt and risky actions

Episodic 6 Allowing Personal events (Detached) memory conscious in particular hierarchically recollections of places and at organized

Musical 7 Facilitating Syntactic (Detached) expectancy symbolic information schematic and language with a hierarchical complex representations semantics

Mechanism Key brain Cultural impact Ontogenic regions and learning development

Brain stem The inferior Low Prior to birth reflex colliculus, the reticulospinal tract of the reticular formation, the intralaminar nuclei of the thalamus

Rhythmic Networks of Low Prior to birth entrainment multiple (perception only) oscillators in the cerebellum and the sensori-motor regions

Evaluative The lateral High Prior to birth conditioning nucleus of the amygdala, the interpositus nucleus of the cerebellum

Contagion ‘Mirror neurons’ Low First year in the pre-motor regions, right inferior frontal regions, the basal ganglia

Visual imagery Spatially mapped High Pre-school years regions of the occipital cortex, the visual association cortex, and (for image generation) left temporo- occipital regions

256 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Episodic The medial High 3–4 years memory temporal lobe, especially the hippocampus, and the dorsal medial prefrontal cortex

Musical The left High 5–11 years expectancy perisylvian cortex, ‘Broca’s area’, the dorsal region of the anterior cingulate cortex, the orbital fronto-lateral cortex

Mechanism Induced affect Temporal focus Induction speed Degree of of affect volitional influence

Brain stem General arousal, Present High Low reflex surprise

Rhythmic General arousal, Present Low Low entrainment feelings of communion

Evaluative Basic emotions Present High Low conditioning

Contagion Basic emotions Present High Low

Visual imagery All possible Omnidirectional Low High emotions

Episodic All possible Past Low Medium memory emotions, but especially nostalgia

Musical Interest, anxiety, Present/Future Medium Low expectancy surprise, chills, hope, disappointment

Brain stem Low High Medium reflex

Rhythmic Low High Medium entrainment

Evaluative Low High Low conditioning

Contagion Low High Medium

Visual imagery High Low Medium

Episodic High Low Low memory

Musial Medium Medium High expectancy

Selected cast and sound crew credits, listed on IMDb as In the Cut (cast & crew) (In the Cut (cast & crew) 2003):

Director: Jane Campion Screenplay: Jane Campion and Susanna Moore

Cast: Meg Ryan Frannie Avery Jennifer Jason Leigh Pauline Mark Ruffalo Detective Giovanni A. Malloy Nick Damici Detective Ritchie Rodriguez

Sound Department: Ken Ishii Production sound mixer Schavaria Reeves Boom operator Andrew Plain Supervising sound editor Linda Murdoch Dialogue editor Peter Miller Sound designer / sound effects and atmospheres editor Mark Ward Sound effects and atmospheres editor Blair Slater Foley supervisor Mario Vaccaro Foley artist Martin Oswin Sound re-recording mixer

Music Department: Hilmar Örn Hilmarsson Composer (original music)

------this page intentionally blank ------

260 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Appendix D: Audiovisual reference materials See accompanying DVD-ROM for supplementary audiovisual materials.

Hovering in the background of this thesis is a certain irony in attempting to describe and analyse the embodied meaning of sound through the written word. When discussing specific audiovisual sequences, text-based analysis abstracts away from materiality of the imagery. This presents a serious obstacle to any analysis of cinematic sound design because the perceptual information which is at the centre of analysis is almost completely lost to the reader except through a second-hand translation.

For this reason, the audiovisual works referenced in this thesis are collated on an accompanying DVD-ROM as individual media files. These media files preserve, where possible, the original soundfield information (e.g. mono, stereo, surround) at the highest resolution available.

List of media excerpts (in title alphabetical order):

A Letter to Three Wives (Mankiewicz 1949) – Sonovox examples Around the World (Daft Punk) (Gondry 1997) Au Clair de la Lune (Scott de Martinville 1860) Babel (Inárritu 2006) Come and See (Klimov 1985) Come into My World (Gondry 2002) Dickson Experimental Sound Film (Dickson 1894/1895) Engine (Cunningham 1999) Gravity “I've Got You” (trailer) (Cuarón 2013a) Huarache Lights (Knowles 2015) In the Cut (Campion 2003) King Kong (Jackson 2005) La Sortie de l’Usine Lumière à Lyon (Workers Exiting the Factory) (Lumière, L 1895)

261 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Lamp (Jonze 2002b) Making of Star Guitar (Gondry 2001a) Mon Oncle (Tati 1958) My Heart Will Go On (Woodruff 1997) Punch-Drunk Love (Anderson, PT 2002) Saving Private Ryan (Spielberg 1998) Second Bad Vilbel (Cunningham 1996) Star Guitar (Gondry 2001b) Sucker Punch (teaser) (Snyder 2011a) Sucker Punch (trailer) (Snyder 2011b) The definition of an emotional roller coaster (Thatcher 2009) The Girl with the Dragon Tattoo (teaser) (Fincher 2011) The Ring (Verbinski 2002) The Sound of Inception (Coleman 2011) Timecode (Figgis 2000) We Are Your Friends (trailer) (Joseph 2015a)

E.1 Components of a soundtrack (DME stems) Typically, the soundtrack possesses a tripartite structure: a) dialogue, b) music, and c) (sound) effects. This is often referred to as the DME format. Each of these three strands of sound – technically referred to as stems – may be further decomposed into sub-categories.

Dialogue Dialogue has four sub-categories: i) sync dialogue ii) wild lines iii) post-sync dialogue (also known as ADR) iv) loop group

Sync dialogue is speech recorded on location synchronously with the camera.

Wild lines are recordings of dialogue on location without the camera.

Post-sync dialogue is dialogue recorded in a studio to picture playback. Post-sync dialogue is also known as ADR, or Automated Dialogue Replacement.

Loop group is also dialogue recorded in a studio but it is not featured dialogue as is the case with sync and post-sync dialogue. Rather, loop groups provide background vocal atmosphere of scene locations. To that end, actors are voice cast in specific idioms to represent onscreen socio-cultural environments. Unlike sync and post-sync dialogue, loop group is speech not intended to be understood as true speech or conversation. Loop group is known as ‘walla’ in the United States.

Source music emanates from within the storyworld, such as music playing from a radio or an iPod. Source music is also known as diegetic music.

Score is music which emanates from outside the storyworld, commenting upon it. Score music is also known as extradiegetic music.

Sound effects Sound effects has three sub-categories: i) Effects ii) Foley iii) Atmospheres

Effects are often described as ‘hard effects’. That is, sounds are literally cut hard into sync with the picture. Effects usually relate to an object or action within the visual frame. Effects are also referred to as SFX.

Foley is sound recorded to picture-playback, and is highly action- and picture-specific. Foley usually refers to body moves (such as clothing, footsteps) and other action which requires fine, naturalistic texture. Foley is also a technique to record unusual or non-naturalistic sounds. Foley is typically used to replace environmental sound effects lost during the ADR process. Foley builds the concrete reality of a character’s presence.

Atmospheres are extended soundscapes specific to a spatial location. Atmospheres are generally comprised of multiple layered sound, for example, traffic echoing down the canyon between city skyscrapers, or a storming ocean crashing against a rocky bluff. The sources of the sounds comprising an atmosphere are often unseen. Atmospheres provide a scene’s sonic environment, sits beneath dialogue, effects and music, and acts to blend these strands together into a unified soundtrack. Atmospheres have the function of cuing the audience to time and place and, most importantly, the emotional range (mood) of the storyworld and/or characters.

Blurring sonic categories In contemporary sound design, the boundaries between the categories of dialogue, music and effects are highly porous. Sometimes voice may cease to be dialogue but become an effect and, at other times, it may become musical. Similarly, an effect may be rhythmic and musical while music may function as an effect.

E.2 Phases of soundtrack design and production Broadly speaking, soundtrack design and production occurs across multiple phases:

Location Location recording (recordist and boom-swinger/s)

Post-production Post-sync dialogue recording (‘ADR’ and ‘loop group’) Foley recording Music recording Editing (dialogue, effects, Foley, atmospheres, music) Mixdown (DME stems premixing, and final mixing)

Exhibition Miscellaneous exhibition format production (e.g. cinema, broadcast TV, Blu-ray, VOD) International language re-versioning

------this page intentionally blank ------

Mark Shelton Ward entered undergraduate studies in film, creative writing, and English, American and Australian literature at the Western Australian Institute of Technology (now Curtin University) in 1979.

In 1986, he graduated from the Australian Film, Television and Radio School (AFTRS) in Sydney, Australia, with a Bachelor of Arts: Film and Television specialising in Directing with recognised work in Sound.

In 1987, he co-founded the sound design house Counterpoint Sound, and in 1993 co-founded the production company Oracle Pictures.

From 2000 until 2006, he worked as a freelance sound designer, and has a wide range of production credits including such Australian productions as Ray Lawrence’s Lantana (2001), Russell Mulcahy’s Swimming Upstream (2002) and Jeff Balsmeyer’s Danny Deckchair (2002), as well as such international productions as Phillip Noyce’ The Quiet American (2001) and Jane Campion’s In the Cut (2003).

Between the years 2006 and 2009, Mark was the Head of Sound at the Australian Film, Television and Radio School, and since that time has continued to develop a broad profile as a media educator lecturing at the University of Technology, Sydney, the University of New South Wales, and Western Sydney University.

In 2005, he commenced doctoral research at University of Technology, Sydney under the supervision of Prof. Ross Gibson, and transferred to University of Canberra to complete the degree under Prof. Gibson’s supervision.

Author’s permanent address: Hazelbrook, New South Wales, Australia.

------this page intentionally blank ------

Aspects of this doctoral research have been previously published and/or presented as:

Ward, MS 2015, 'Art in noise: an embodied simulation account of cinematic sound design', in M Coëgnarts & P Kravanja (eds), Embodied cognition and cinema, Leuven University Press, Leuven, Belgium, pp. 155–86.

Ward, MS 2013, Art in noise: an embodied simulation account of cinematic sound design, paper presented at the annual conference of the Society for Cognitive Studies of the Moving Image (12–15 June 2013), Universität der Künste Berlin.

Ward, MS 2010, 'Voice, videogames, and the technologies of immersion', in N Neumark, R Gibson & TV Leeuwen (eds), Voice: vocal aesthetics in digital arts and media, MIT Press, Cambridge, Mass., pp. 267–79.

Ward, MS & Leung, L 2008, 'Beyond the visual: applying cinematic sound design to the online environment', in L Leung (ed.), Digital experience design: ideas, industries, interaction, Intellect, Bristol, UK, pp. 81–91.

Drago, C, Leung, L & Ward, MS 2008, 'What's the story? Harnessing the power of storytelling in film for Experience Design', in L Leung (ed.), Digital experience design: ideas, industries, interaction, Intellect, Bristol, UK, pp. 35–47.

------this page intentionally blank ------

1 Lawrence Shapiro describes embodied cognition as less a theory than a research programme unified by its commitment to “elevate the importance of the body in the explanation of various cognitive abilities” (Shapiro 2007, p. 340). 2 For examples, see Sobchack (2004) and Elsaesser & Hagener (2010a). 3 My reference to theories of affect should not be confused with ‘affect theory’ currently in use within humanities discourse. My use of the term affect is not identical to its use by affect theorists such as Brian Massumi (2002a), Nigel Thrift (2004; 2008), or William Connolly (2002). For clarification of my use of the term see Chapter 4. 4 In this thesis, the terms ‘film-sound’ and ‘cinematic sound’ are used somewhat interchangeably. However, there are restrictions. Film-sound typically denotes classical theories of cinematic sound, and are bound by that era’s focus upon the materiality of film. Cinematic sound, on the other hand, is not restricted to the medium of film but may be found wherever cinematic imagery operates. 5 See Philip Brophy (2008, p. 434) for an iconoclastic, if idiosyncratic, discussion of film theory’s ‘absent aural’. 6 My curriculum vitae appears in Appendix E. 7 For a discussion of ocularcentrism and post-Socratic philosophy see Levin (1993b). 8 In detailing this premise I follow the merits of an ecological and evolutionary approach to film theory laid out by Anderson & Anderson (1996; 2005). 9 For a definition and discussion of core affect see Russell (2003, 2005, 2009). 10 For a discussion of audiovisual emotion and meta-emotion see Bartsch (2004, 2005). 11 For a discussion of the cinema as an ‘invitation to feel’ see Smith (2003c) 12 It is somewhat common to find the terms nonconscious, unconscious and subconscious used interchangeably. However, in this thesis my preference is for the term nonconscious. This is intended to avoid confusion with notions of the Freudian ‘unconscious’ and ‘subconscious’. In this way, I follow the contemporary neurocognitive literature in acknowledging a distinction between conscious and nonconscious processing (LeDoux 2000), in nonconscious mechanisms which govern behaviour (Bargh & Chartrand 1999), and in mechanisms of nonconscious emotion (de Gelder 2005; Kihlstrom et al. 2000). My choice of term is also intended to align with Damásio’s definition of the proto-self as a state of nonconsciousness which is the necessary foundation upon which consciousness proper emerges (Damásio, AR 1999, 2010). It should be noted that Lakoff employs the term ‘cognitive unconscious’ yet goes to some length to distance the term from its Freudian overtones (Lakoff & Johnson 1999). 13 See Chapter 3 for a definition of ‘bottom-up’ and ‘top-down’ processing. 14 Embodied simulation theory is presented in detail in Chapter 5. However, for a definition and discussion of embodied simulation theory and its relationship to intercorporeity, intersubjectivity and empathy see Gallese (Gallese 2001, 2003b, 2005, 2006, 2007a; Gallese, Eagle & Migone 2007; Gallese & Sinigaglia 2011a). 15 Neuroaesthetics is also sometimes referred to as empirical aesthetics. 16 Other important figures include philosopher William Hirstein particularly in collaboration with Ramachandran (1999), John Onians (2007, 2011), Suzanne Nalbantian (2008), Cinzia Di Dio and Vittorio Gallese (2009), Emily Cross and Luca Ticini (2012), and Elvira Brattico and Marcus Pearce (2013). 17 Raymond Tallis has coined the term neuromythology (1999, 2011) to describe, albeit derisively, a certain tendency in appealing to neuroscience when accounting for problems the humanities historically concern themselves but have found intractable. 18 Related to Smith’s mood-cue approach is Torben Grodal’s PECMA flow model of filmic experience (Grodal 2006, 2009c). As the name suggests, the PECMA model privileges the interaction of perception, emotion, cognition and motor action as a central mechanism in explaining cinematic experience, and does so within an evolutionary framework. The PECMA model, from my perspective, is powerful but limited by a latent ocularcentrism and narrativism. Smith’s mood-cue approach and Grodal’s PECMA flow model are related in that both recognise and prioritise the importance of emotion in generating cinematic meaning. 19 Thus far my professional practice spans 30 years as a sound designer in the globalised Australian cinema and television industry. 20 Commencing with C.P. Snow’s call to action in his article in the New Statesman (Snow 1956), the first wave of consilience can be said to arise with E.O. Wilson’s Consilience: the unity of knowledge (Wilson, EO 1998), while Edward Slingerland and Mark Collard (2011b) note the emergence of a second wave. 21 For a definition of my use of the term cinematic see Chapter 3, and the terms cinematic imagery and cinematic media in the glossary. 22 The relationship between body and consciousness is explored in detail in Chapter 7. 23 Connected to the assumption which values the mental over the physical are the concepts of Verstehen (interpretive understanding) and Erklärung (causal explanation) which are considered to divide the forms of significance typically generated within the humanities and sciences respectively. For an explication of the concepts of Verstehen and Erklärung, and their relevance to the tension between the conventional stances taken up within the humanities and sciences, see Slingerland (2008a), particularly pages 2–4 and 83. For a consideration of embodiment as an approach by which to bond mind and body, Verstehen and Erklärung, see Caracciolo (2014). 24 This thesis recognises both visual and auditory forms of imagery. For a review of the empirical literature on auditory imagery see Hubbard (2010). 25 However, voice may also be genuinely claimed to be a designed sound element. For a discussion of how cinematic voices are designed see Macallan & Plain (2010) and Pauletto (2012). 26 Under this definition, television, computergames, virtual reality (VR), and even music and graphic novels, may qualify as ‘cinematic’ media.

27 While the terms film-sound, cinema sound and cinematic sound may appear interchangeable I ascribe slightly different connotations to each. For example, film-sound emerges as a term from theories of sound emanating from classical cinema. Cinema sound updates film-sound within a contemporary post-classical context. Both film-sound and cinema sound, however, are technologically anchored. In contrast, my use of the term cinematic sound foregrounds qualities of sonic imagery which are technologically agnostic and transposable across media. 28 Altman, for example, argues for a radically broadened definition of ‘film sound’. Under his enhanced definition – which moves away from the “overly narrow, inherited as it is from a text-oriented tradition” (1999a, p. 31) – Altman includes four sonic categories: sound of the film, audience sounds, noise of the apparatus, and advertising sound (1999a, p. 32). 29 In addition to the work of Grimshaw and Garner on embodied acoustic virtual space is the edited volume Body, sound and space in music and beyond: multimodal explorations by Clemens Wöllner (Wöllner 2017). 30 Of course, since David Yewdall’s and Vanessa Ament’s writings on the practice of Foley (Ament 2009; Yewdall 2003), it is important to note the recent scholarly articles published on the subject (Donaldson 2014; Hagood 2014; Lewis, M 2015; Wright 2014). 31 Indeed, Doug Kahn critiques Cage for ‘musicalising’ noise to imbue it with readymade significance (1999, p. 18). 32 An example of de Martinville’s phonautograph, a recording of Au Clair de la Lune (Scott de Martinville 1860) is available on the accompanying DVD-ROM to Appendix D. 33 Indeed, de Martinville fought a public campaign against what he considered to be Edison’s incorrect and non-authentic use of the concept behind the phonautograph. 34 Historiographies of technological developments often suggest these technologies spring fully formed from the minds of their inventors. But this is an error. Developments in technology do not arise as single moments of great change but usually emerge from a continuum and almost always from group effort. The idealising of a single originator should be resisted. See Appendix A for comparative timelines illustrating the interwoven nature of audiovisual technological development. 35 For example, La Sortie de l'Usine Lumière à Lyon (Lumière, L 1895). 36 For an overview of the intellectual development of the ‘narrative turn’ as it expanded from literary theory through the humanities and social sciences see Kreiswirth (1992, 2005) and Czarniawska-Joerges (2004). See Hyvärinen (2010) for a discussion of the four phases of the ‘narrative turn’. 37 See Fox (1980) for an interview with Murch discussing the concept and practice of ‘sound design’ soon after the release of Apocalypse Now. See also Jarrett (2000) for an interview with Murch discussing the concept after two decades. 38 For a discussion of sound designer as composer, see Keane (2007). 39 Compare my definitions with those of William Whittington who proposes sound design may be defined in four ways: i) the “creation of specific sound effects”; ii) the “conceptual design of the overall sound track”; iii) the “deployment of sound within the theatrical space” (such as via multichannel exhibition); and iv) “a model and method for critical analysis” which integrates the previous three definitions as a conceptual model (Whittington 2007, pp. 2-3). Thus Whittington arrives at a definition where “sound design represents the planning and patterns of the film sound track and the meanings that result from its deployment within the exhibition space” (2007, p. 3). 40 American Zoetrope is the San Francisco-based production company established by Francis Ford Coppola and George Lucas in creative response to European arthouse cinema and envisioned as an alternative model to Hollywood-based movie production (Whittington 2009, pp. 557-558) 41 There are three sets of Sonovox sounded flashbacks in A Letter to Three Wives (Mankiewicz 1949) The first occurs at 00:12:23–00:13:20 and 00:31:20–00:31:50 (hh:mm:ss). The second occurs at 00:33:30–00:34:05 and 00:58:40–00:59:20 (hh:mm:ss). The third occurs at 01:00:45–01:01:35 (hh:mm:ss). A compile of these transitions is provided in the supplementary audiovisual material of Appendix D. 42 Walt Disney’s animated feature Dumbo (Sharpsteen 1941) predates both Possessed (Bernhardt 1947) and A Letter to Three Wives (Mankiewicz 1949) in its use of the Sonovox. See Jacob Smith (2008) for an analysis of the Sonovox in cinema, in particular its relationship to the female voice and melodrama. 43 The commercial transition to sound-film is customarily associated with the release of The Jazz Singer (Crosland 1927). For an exploration of sound design in Classical Hollywood see Hanson (2007). 44 For a discussion of the radio aesthetic in cinema, particularly with regard to Orson Welles and Citizen Kane (Welles 1941), see Altman (1994). 45 Indeed, in the North American context, it is Nichols and her husband who are credited with importing the knowledge of sound effects from vaudeville into radio in the late 1920s (Mott 1993, p. 19). 46 See Horning (2004) for a discussion of the early sound recording engineers’ “unarticulated, implicit knowledge gained from practical experience” (Horning 2004, p. 707). 47 For discussion of the many and varied sound practices developed for silent cinema see Richard Abel and Rick Altman (Abel & Altman 2001; Altman 1996, 2004, 2012). 48 What I describe here relates to feature drama movies but is applicable to documentary. For a brief but insightful discussion of documentary sound design see the interview with Coll Anderson (Coleman 2015), sound designer of such Errol Morris’ documentaries as Fog of War (2003) and Mr. Death: The Rise and Fall of Fred A. Leuchter, Jr. (1999). 49 An example of this hand-in-glove process is Murch’s own role as both picture editor and sound designer of Apocalypse Now and picture editor and re-recording mixer of The English Patient (Minghella 1996). 50 For further critique of the notion of ‘sound designer’, albeit limited, see Beck (2008, pp. 73-76). For an extensive description of the development of the concept of ‘sound design’ and its cultural and political dimensions and confusions, see Wright (2013) and Whittington (2007, pp. 24-27). 51 It is interesting to note Pauletto’s (2012) interest in the voice as a sound-designed artefact resides in the crafting of acoustic properties not the verbal content of speech. 52 Gunning (Gunning 1986) suggests early cinema can be said to arise around 1895 and came to a close with its

272 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. narrativisation around 1919 while Bordwell and Thompson (Bordwell, Staiger & Thompson 1985) suggest classic Hollywood arose around 1917 and came to a close around 1960. 53 For an overview of the life, work and film theoretical legacy of Münsterberg see Sinnerbrink (2009). 54 For a discussion of Münsterberg’s approach to cinematic processes as if mental processes, and of the film/mind analogy generally, see Noël Carroll (1988). 55 For a wonderfully detailed history, description and legacy of the Allefex and similar sound effects machines see Bottomore’s ‘An international survey of sound effects in early cinema’ (1999). 56 See Giuliana Bruno (Bruno 2009) and Robert Brain (Brain 2012) for a discussion of Münsterberg’s development of an experimental aesthetics. 57 In attendance at the Brighton conference were André Gaudreault (Gaudreault 1983, 1987, 2009; Gaudreault & Gunning 1989), Tom Gunning (Gunning 1986, 1989, 1991, 1993, 1995), Charles Musser (Musser 1991, 1994b, 1994a), Eileen Bowser (Bowser 1994), Michael Chanan (Chanan 1980), and Barry Salt (Salt 1978, 1984), each of whom emerged as prominent contemporary theorists of early cinema. 58 See Gaudrault’s ‘From "Primitive Cinema" to "Kine-Attractography”’ (Gaudreault 2006) for an analysis of the relative limitations of the terms ‘primitive cinema’ and ‘early cinema’. 59 Gunning cites Donald Crafton’s ‘Pie and chase: gag, spectacle and narrative in slapstick comedy' (Crafton 1995) for its examination of slapstick as the site where spectacle and narrative are first blended. 60 However, note The story of the Kelly Gang (Tait 1906), listed in the UNESCO Memory of the World Register as the world's first full-length narrative feature film, predates Gunning’s periodisation. 61 For a detailed study of these epochs from an American perspective see Charles Musser’s The emergence of cinema: the American screen to 1907 (Musser 1994b), Eileen Bowser’s The transformation of cinema, 1907-1915 (Bowser 1994), and Richard Koszarski’s An evening's entertainment: the age of the silent feature picture, 1915-1928 (Koszarski 1994). 62 Wanda Strauven has traced the term’s Byzantine iteration over the period from 1985 to 1990 (2006a, p. 14). I shall not recount this development except to say the concept’s genesis appears to stem from a co-authored paper Gaudreault and Gunning presented at the Cerisy Colloquium on film history in August 1985 and published in 1989 as ‘Le cinéma des premiers temps: un défi a l'histoire du cinéma?’ (‘Early cinema as a challenge to film history’) (Gaudreault & Gunning 1989). However, the term was not explicitly defined until Gunning’s ‘The cinema of attraction: early film, its spectator and the Avant-Garde’ (Gunning 1986). 63 Eisenstein wrote ‘Montage of attractions, an essay' (Eisenstein 1947 [1923]) as he was finishing his first feature film, Strike (Eisenstein 1925). Eisenstein grasped the radical nature inherent in a cinema based upon attractions, a cinema which addressed itself directly to its audience, and desired to harness this power for politically revolutionary ends. 64 On this point, Gunning notes “recent spectacle cinema has re-affirmed its roots in stimulus and carnival rides, in what might be called the Spielberg-Lucas-Coppola cinema of effects” (Gunning 1986, p. 70) indicating a link between the early and post-classical cinema that elides narrative for other forms of audience engagement. 65 Domitor is an association founded in 1985 by Stephen Bottomore, Paolo Cherchi Usai, André Gaudreault, Tom Gunning and Emmanuelle Toulet for the study of early cinema from its origins to 1915 (Domitor). As the association’s website attests, “Domitor is not an acronym: it revives the name that the father of the Lumière brothers once proposed for their projector of motion pictures.” 66 Elements of the conference were initially edited by Abel and Altman as a special issue for the journal Film History entitled ‘Global experiments in early synchronous sound’ (‘Global experiments in early synchronous sound’ 1999), but Abel and Altman edited the proceedings soon after as The sounds of early cinema (Abel & Altman 2001), a text which remains one of the most significant on the subject of sounded early cinema. 67 See Loughney(1999) for a discussion of how the Dickson Experimental Sound Film came to be restored and presented at the 1998 Domitor conference. The Dickson Experimental Sound Film (Dickson 1894/1895) is available on the accompanying DVD-ROM to Appendix D. 68 For a detailed exploration of Hollywood and Europe’s transition to synchronous sound see Donald Crafton’s The talkies: American cinema's transition to sound, 1926-1931 (Crafton 1997), Scott Eyman’s The speed of sound: Hollywood and the talkie revolution, 1926-1930 (Eyman 1999), and Charles O' Brien’s Cinema's conversion to sound technology and film style in France and the U.S. (O' Brien 2005). 69 Belton, for example, argues the “design of early sound recording technology played a crucial role in determining the “look” of early sound films” (Belton 1999, p. 228), and Whittington (Whittington 2007, p. 131) notes how sound was a major tool for maintaining cohesion and tension in early sound horror films, a genre that relied not upon a frame of narrative causality and unity for understanding but an affective frame. 70 Indeed, a brief survey of the selected essays collected under the heading ‘classical sound theory’ in Elisabeth Weis’ and John Belton’s touchstone volume Film sound: theory and practice (Weis & Belton 1985) reveals a deep antipathy toward synchronous film; that is to say, synchronous dialogue. 71 Undoubtedly, Russolo’s dream of noise-sounds was radical and difficult to comprehend for those who did not share his Futurist perspective, but in his writing I suggest we can detect something at work beside merely the modernist ‘shock of the new’. Throughout the 20th century, Russolo’s dream would coalesce and evapourate, appearing intermittently in the various works of Edgard Varèse, Pierre Schaeffer, Iannis Xenakis, and John Cage who, each in their way, would enact the idea of a music based upon noise-as-music. However, it was to take a much more radical shift in paradigm to bring about Russolo’s art of noise. Arguably, an art of noise would not fully come into its own until 1979 when Peter Vogel and Kim Ryrie unveiled the Fairlight CMI, a ‘computerised musical instrument’ capable of digitally sampling and modelling sound. Although I turn away from the ideology of Futurism, I share Russolo’s suspicion of the sedimented formulations and restrictions placed upon sound to be made culturally accessible (and acceptable) as ‘music’. As a result, it is not music nor voice per se that interests me in this thesis, but the wider sounding universe, and how that universe may be modelled and manipulated within cinematic

273 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. sound design. 72 Two decades later, in his 1999 article ‘Sound Studies: a field whose time has come’ (Altman 1999c), Altman observes the field’s birth in cinema and its blossoming into a “a decidedly interdisciplinary field” (Altman 1999c, p. 4). Sound Studies, however, has a long period of gestation. Mattern & Salmon, for example, argue it can be traced to at least John Cage and his time at The New School (Mattern & Salmon 2008). 73 Schaeffer’s compositional approach sought to loosen itself from adherence to strict conventional musical structure of melody, harmony, rhythm, and metre. 74 Importantly, Schaeffer’s mode of reduced listening underpins his concept of the sound object in musique concrète. 75 There are other taxonomies of listening. William Gaver (Gaver 1988, 1993a, 1993b) and David Huron (Huron 2002), for example, have independently developed taxonomies in the fields of human-computer interaction and music cognition, respectively. Tuuri and associates (Tuuri & Eerola 2012; Tuuri, Mustonen & Pirhonen 2007) have built upon Chion (and thus Schaeffer) by detailing nine listening modes (reflexive, kinaesthetic, connotative, causal, empathetic, functional, semantic, reduced, and critical listening). For a comparison of the taxonomies developed by Schaeffer, Chion, Gaver, Huron, and Tuuri see Caramiaux et al. (Caramiaux et al. 2014, pp. 35–37) and Vickers (Vickers, P 2012). 76 Audio-vision: sound on screen (Chion 1994) systematises the many concepts and neologisms drawn from his earlier three works La Voix au cinéma (Chion 1982), Le Son au cinéma (Chion 1985), and La Toile trouée (Chion 1988). 77 Additionally, it is worthwhile noting Chion’s claim the soundtrack lacks internal coherence is not borne out by professional sound practice. It is not uncommon, for example, for sound editors to work for periods of time on a scene without reference to the visual image. The specific purpose of this technique is to build internal coherence and balance within the soundfield, and this practice of prioritising sound-sound relationships extends to re-recording mixers. What these sound designers are doing as they work ‘blind’ is shape the internal structure, coherence, flow and meaning of the soundtrack. 78 It is of interest to note Bregman has published research with Poppy Crum (Chief Scientist at Dolby Laboratories) on the perceptual system’s capacity to not only detect sound unit boundaries but to process within those units differently (Crum & Bregman 2006). 79 Having said this, it seems Chion’s view of what he calls ‘Dolby sound’ has softened from suspicion to interest (Chion 2003; Chion & Brewster 1991). 80 Contemporary VR research has shifted toward a focus on perceptual realism to reveal the design of sound is powerfully implicated in inducting a subjective sense of immersion and presence (Väljamäe et al. 2008), and the closing of the perceptual loop that interaction affords builds upon traditional cinematic design techniques. 81 Indeed, Walter Murch admits to being personally inspired by musique concrète, and he notes parallels between Shaeffer’s approach and the early experiments in sound montage undertaken by Welles and others in radio and cinema (Jarrett & Murch 2000, p. 6). 82 For an overview of the significance of Schaeffer’s work to the practice and theory of audiovisual composition – and Chion’s point of difference with Schaeffer over the notion of counterpoint – see Bizzaro (2011). 83 Thanouli has shown post-classical cinema to be not just a label applicable to only Hollywood product, but may be applied to movies produced in the USA more generally as well as Europe and Asia (Thanouli 2006). 84 See also Stork’s polemicised through a series of audiovisual essays (Stork 2011b, 2011c, 2011a). 85 However, these characterisations are contentious. For example, Sara Ahmed (Ahmed 2010) has critiqued the claim affect is separable from emotion as gendered. 86 So numerous are accounts of the trans-sensory that Tarja Laine and Wanda Strauven propose the humanities, arts, sciences and technology are in the midst of a ‘synaesthetic turn’ (Laine & Strauven 2009). Indeed, reflecting the current era’s preponderance for theorising the sensory and somatic, Elsaesser & Hagener structure their historiography Film theory: an introduction through the senses (Elsaesser & Hagener 2010b) via the use of these dimensions as both organising principle and heuristic; for example, ‘cinema as window’, ‘cinema as eye’, ‘cinema as ear’, ‘cinema as skin’, and so on. 87 See Paul Elliott (Elliott 2011, p. 32). 88 This understanding – that bodily representations underpin the functioning of metaphor and thought – foreshadows the concept of B-format (bodily format) representations developed by Alvin Goldman and Frederique de Vignemont (Goldman 2014; Goldman & de Vignemont 2009) and the neural re-use theory developed by Michael Anderson (Anderson, ML 2010). B-formatted representations are interoceptive in nature and herald from the simulation theory of social cognition developed by Vittorio Gallese and Alvin Goldman (Gallese & Goldman 1998). Like Sobchack’s proposal, these hypotheses present a multilevel system of meaning-making that spans from the autonomic to the conscious but has its basis in the human body. 89 Indeed, Barker’s proposition of a sensorimotor mirroring induced in an audience by cinematographic movement has been empirically substantiated by Gallese and colleagues (Heimann et al. 2019; Heimann et al. 2014). 90 Musing upon a Film Studies that might have been, Nick Refern (Redfern 2011) points to how early empirical film research was being undertaken at exactly the same time as the hegemony of contemporary 1970s film theory was being established, suggesting how Film Studies could travelled down a different path. 91 Dissatisfaction with Theory did, of course, occur earlier. Raymond Durgnat’s ‘The death of cinesemiology (with not even a whimper)’ (Durgnat 1980) and ‘The language of film and problems in cinesemiotics’ (Durgnat 1982), for example, foreshadowed Bordwell and Carroll’s project to loosen the hold of Theory upon Film Studies. 92 See, for example, Lisa Zunshine’s edited volume Introduction to cognitive cultural studies (Zunshine 2010), Joan Chiao’s work in cultural neuroscience (Chiao 2009a, 2009b; Chiao & Ambady 2007) and the edited volume by Shihui Han, et al. (Han et al. 2013). 93 See, for example, Nannicelli & Taberham’s Cognitive media theory (Nannicelli & Taberham 2014). 94 Aside from Jeff Smith’s discussion of music score (Smith, J 1999) and Greg Smith’s inclusion of sound generally within his mood-cue approach (Smith, GM 1999), sound design and its relationship to emotion and cognition is under-explored in this volume.

95 Kerins observes many scholars of film-sound possess “a limited understanding of the various processes and people involved in the creation of motion picture soundtracks” and “[t]his level of ignorance about actual production strategies and techniques results in incomplete, and possibly inaccurate, analyses” (Kerins 2008, pp. 116-117). To remedy, Kerins suggests greater effort should be expended in primary research which is also grounded in understanding the praxis of media-making so as to explain how “decisions about soundtracks – like those about most elements of filmmaking – are hybrids of creative impulses, technical limitations, and cultural preferences” (Kerins 2008, p. 117). 96 Kerins’ wish is to “find ways to expand analysis beyond music and musicality, in establishing a collective vocabulary that moves beyond the classical terms of music to encompass the rich variety and texture of sounds used in the cinema and other audiovisual media” (Kerins 2008, p. 118). In recognising there are distinctions to be made between music, sound and soundtrack, Kerins acknowledges the limitations of critical terminology imported from other fields such as musicology and cultural studies, and is critical of cinema sound scholarship which “has tended to privilege the music-driven elements of the soundtrack” (Kerins 2008, p. 118). Like Altman (Altman 1992a), Kerins maintains musical concepts and terminology “go only so far in analysing the [soundtrack’s] many extra-musical sonic elements” (Kerins 2008, pp. 117-118). This insight echoes my own assumption of music to be only a fraction of the possibilities of a sounding universe. The negative consequences of a music-centred terminology, Kerins notes, is a reduced capacity to describe and analyse spatiality and presence, and to which I further add the concepts of spectacle, intra- and crossmodal interaction and ecological psychoacoustics. 97 Kerins values the possibilities of interdisciplinary research to produce an integrated approach to soundtrack analysis, but is wary of a generalising effect that gravitates away from a “truly filmic model of sound studies” (Kerins 2008, p. 115, emphasis in original). 98 Debord’s ideological formulation of spectacle is encapsulated in his assertion that the “spectacle is ideology par excellence, because it exposes and manifests in its fullness the essence of all ideological systems: the impoverishment, the servitude and the negation of real life.” (Debord 1967, paragraph 215) 99 Bordwell suggests these visual tactics to be the most dominant in contemporary cinema although there may be others, such as “axial cut-ins, desaturated and monochromatic color schemes, slow motion, and handheld shooting” (Bordwell 2002, p. 19). 100 The issue of ‘loudness’ turns on the contemporary use of relatively short but higher peak sound levels. See Ioan Allen (1998) and Earl Vickers (2010) for a discussion of what has come to be known as the ‘loudness wars’. 101 Another consequence of the Academy curve is its privileging of instruments and musical styles that share the same frequency spectrum occupied by the voice. 102 The use of LFE, it should be noted, goes largely unnoticed by a general audience, but still registers in the overall feel of a scene, no matter if the scene includes raging dinosaurs or a humble conversation around a dining table. 103 Dolby Atmos (2012) is the latest example of an advanced commercial surround soundfield technology. The Dolby Atmos format supports the processing of up to 128 discrete audio tracks distributed to up to 64 speaker feeds, some of which emanate from above the audience thus providing XYZ spatial dimensions. See Sergi (Sergi 2013) for a discussion of the nature of the immersive audio experience created by the Dolby Atmos platform. 104 For a review of scientific literature concerning ‘musical chills’ see Harrison & Loui (Harrison & Loui 2014). 105 That is to say, cinema-plus is cinema in the conventional sense but which affords the added dimension of agency to act in non-trivial ways within the mediated world, such as in the case of computergames, virtual environments, and persistent online worlds. 106 Yuri Tsivian has proposed a somewhat ‘agnostic’ approach to the definition of cinema suggesting it to be “a time-bound medium with or without narrative motivation” (2008, p. 764), an approach which is more or less in accord with the one I take here. 107 Walter Murch (1992, 2000, 2005) and Randy Thom (1999, 2007, 2009), of course, have consistently presented their ideas about cinematic sound to a wider audience yet they remain decidedly designers of sound rather than writers of theory. Rare examples of practitioner–theorists of sound might include Michel Chion and Daniel Levitin. Chion (1994, 1999, 2009) is a music composer as well as noted film-sound theorist while Levitin (2006) is a music producer who turned to a systematic programme of research in the neuropsychology of music. 108 I first came to publish these assumptions in 2008 (Ward, MS & Leung 2008), and they have gained detail and precision over time (Ward, MS 2010, 2013, 2015). 109 Patrick Colm Hogan goes so far as to say narrative is “fundamentally shaped by our emotion systems” and thus narrative is “inseparable from emotions” (2010, p. 65). 110 For a discussion of the primacy of movement and animate life see Maxine Sheets-Johnstone (1999a). 111 For a critique of Handel’s claims see Michael Kubovy (1988). 112 For reviews of the empirical literature on auditory imagery see Timothy Hubbard (2010) and Daniel Reisberg (1992). 113 For more on perceptual filling-in see Luiz Pessoa (1998). 114 The extent to which sensory mechanisms are interrelated may be illustrated by the kind of neural re-organisation which allows the profoundly deaf to recruit the planum temporale (an area of the right auditory cortex responsible for processing auditory motion) for the processing of visual stimuli (Finney et al. 2003; Finney, Fine & Dobkins 2001), and, conversely, allows the blind to process moving auditory stimuli in the right visual cortex (Weeks et al. 2000). Visual-to-auditory and auditory-to-visual crossmodal plasticity implies a deep biological drive to extract spatiotemporal information from the world. 115 In a similar vein, radio commercials deploying sound effects have been shown to generate strong affective mental imagery (Miller, DW & Marks 1992). 116 Tim Rohrer explains that, “[a]s with the Gestalt perceptual structures, image schemas are dynamic perceptual wholes which prompt for ‘normal’ pattern completions based on our recurrent experiences in interacting with the world” (2011, p. 8).

117 The Jazz Singer (Crosland 1927) is typically considered the commercial watershed between silent and sound-film (Crafton 1997; Eyman 1999). 118 Released in the same year as The Jazz Singer (1927), Sunrise was “shot as a silent film but synchronised with a music score for distribution” (Fischer, L 2003, p. 76). In fact, Sunrise possessed an optical sound-on-film soundtrack comprised of both a music score and sound effects (Miller, P 1982, p. 584). However, most audiences experienced Sunrise as a silent movie because only a few cinemas were fitted with the necessary exhibition equipment. Developed throughout Murnau’s prior films, Sunrise marks the acme of the visual strategy of “sounding images” (Szalóky 2002, p. 117). In a comparable fashion, Adrian Martin notes Fritz Lang’s use of sound to cue visual editing rhythm and drive plot (2000, 2002). For further discussion of cinematic vision structured by an implied diegetic audition see Perron (2001). 119 Many examples of this aesthetic effect may be found in the work of David Lynch, particularly Mulholland Drive (2001), Lost Highway (1997) and Eraserhead (1977). 120 For more on perceptual freezing, and its relationship to the attentional blink, see Olivers & Van der Burg (2008). 121 The teaser for The Girl with the Dragon Tattoo (2011) is available on the accompanying DVD-ROM to Appendix D. 122 See Tierney & Kraus (2013) for a discussion concerning the effect of enhanced cortical and brainstem processing when entraining to a strong musical beat. 123 The trailer Gravity “I've Got You” (Cuarón 2013a) is available on the accompanying DVD-ROM to Appendix D. 124 Interestingly, a similar crossmodal effect cuts in the opposite direction such that auditory intensity (loudness) is enhanced when accompanied by a brief visual event (Odgaard, Arieh & Marks 2004). 125 For definition of the reality effect as it pertains to apparatus and psychoanalytic film theory see Oudart (1990). See also Barthes (1968) who first applied the term to literary texts. 126 Both La Sortie de l'Usine Lumière à Lyon and Dickson Experimental Sound Film are available on the accompanying DVD-ROM to Appendix D. 127 The relevant excerpts from these movies are available on the accompanying DVD-ROM to Appendix D. 128 My Heart Will Go On (Woodruff 1997) is available on the accompanying DVD-ROM to Appendix D. 129 See Carroll’s critique of the medium specificity thesis as it pertains to film theory (1985). See also Maras and Sutton (2000) for an application of medium specificity to new media. 130 For a discussion of the silence of Arnheim’s cinema see Nora Alter (2011). 131 For an illustration of multimodal metaphor see Chapter 7. 132 Intriguingly, the influence of affect in intermodal binding is not limited to the human voice and face, but extends to include spoken and written words (Zeelenberg & Bocanegra 2010; Zeelenberg, Wagenmakers & Rotteveel 2006). 133 As with voice and face, it is important to note this effect is bidirectional such that music influences the perceived affective content in both still and moving pictures, and vice versa (Hanser & Mark 2013, p. 318). 134 In the instance of this research, the soundtrack was non-spatial; that is, monophonic. 135 Notably, the principal sound designer of both Saving Private Ryan and Monsters, Inc. is Gary Rydstrom. Although Rassell and colleagues make this observation, what is left unexplored is the possibility that Rydstrom may demonstrate some form of auteurship with regard to cinematic sound design. 136 Playback of the movie was configured as either sound on or sound off. Soundtracks were the complete DME configuration of dialogue, music and sound effects. 137 Techniques for audience eye-tracking and the visual ‘heat-maps’ produced by such tracking generate much needed data on just what precisely happens when the visual gaze is engaged by moving pictures. However, it should be noted that assessment of the contents of visual attention is much more confidently arrived at than the contents of auditory attention. Thus far there is no auditory equivalent to eye-tracking and visual heat-maps and, because the use of eye-tracking techniques are becoming relatively commonplace, there is the risk that emerging models of multimodal attention will entrench a bias toward the dynamics of visual attention. 138 Excerpts from Timecode (Figgis 2000), the T-rex battle in King Kong (Jackson 2005), and the ‘A Visitor’ sequence in Mon Oncle (Tati 1958) are provided on the DVD-ROM accompanying Appendix D. 139 For a comprehensive review of human multisensory development see Bremner, Lewkowicz & Spence (2012) particularly the chapters ‘Crossmodal interactions in the human newborn: new answers to Molyneux's question’ (Streri 2012) and 'The development of multisensory representations of the body and of the space around the body' (Bremner, Holmes & Spence 2012). 140 Multimodal interaction mechanisms are truly many and varied. For a current and comprehensive review see Calvert, Spence & Stein (2004) and Murray & Wallace (2011). 141 In this regard, it is interesting to note Joseph Anderson’s prescient assessment of crossmodal confirmation as a fundamental mechanism for the creation of cinematic events (1993). 142 Space does not allow a detailed exploration of all synaesthetic effects relevant to cinematic imagery, but for a review of contemporary empirical research into synaesthesia see Jamie Ward (2013; 2006). 143 Traditional magnetic film techniques are limited to editing in increments of a single frame. Cinema conventionally runs at 24 frames per second and thus one frame is 0.0416 second in duration. Digital techniques permit the editing in increments of 0.001 second in duration. 144 Van Wassenhove, Grant & Poeppel (2007) identify this temporal window as holding for audiovisual speech but it may be assumed to extend to other ecologically valid audiovisual stimuli. Kawachi, Grove & Sakurai, for example, note the human perceptual system is capable of using a single, simple auditory cue to resolve multiple visual events such that ‘bouncing’ disks in a bounce/stream experiment may be integrated within a temporal window “offset 60 ms before and up to more than 120 ms after the audiovisual coincidence” (Kawachi, Grove & Sakurai 2014, p. 10 ). 145 For an example of audiovisual metaphor, see Chris Cunningham’s TVC Engine (1999) available on the accompanying DVD-ROM to Appendix D.

146 Also, for a detailed examination of audiovisual metaphor, see my analysis of Chris Cunningham’s Engine (Cunningham 1999) in Chapter 7. 147 Digital ‘chunkiness’ is also a dominant visual aesthetic within the virtual world of TRON, acting to contrast with the smoothness of the real world. This crystalline aesthetic grounds the concepts of risk and threat made manifest in the combat sequences where characters literally disintegrate into blocks of digital debris. This digital aesthetic also underpins the narrative’s notions of racial and social identity. In this regard, it is curious to note the parallel with The Matrix (Wachowski & Wachowski 1999) where Neo’s voice similarly disintegrates into digital granularity at the point of his first passing from the world of the Matrix into the real world. In this instance, the disintegration of Neo’s voice is synonymous with the disintegration of his (digital) self. 148 For an explication of conceptual blending theory see Gilles Fauconnier and Mark Turner (1985; 2002). For an illustration of conceptual metaphor and blending as it operates within Sino-Tibetan language see Edward Slingerland (2005). For an examination of conceptual metaphor and blending operating within visual commercial advertising see Joy, Sherry & Deschenes (2009). For a review of the significance of conceptual blending theory to the history of emotions see Cristóbal Pagán Cánovas (2010). 149 DAWs provide an enhanced ability to layer sound elements into complex patterns without deterioration in signal-to-noise ratio. 150 Parallel with the increasing significance of timbre in cinematic sound is Rebecca Leydon’s observation that contemporary music is “increasingly focused on timbre as a crucial semantic feature” (2012, p. 1), and argues the urgent need for a theory of timbre to explain its function. 151 This design impulse is traceable from contemporary sound design practice back through the work of Walter Murch (Coppola 1979) to Murray Spivack (Cooper & Schoedsack 1933) and the practitioners of sound effects in radio and the silent cinema such as Ora Nichols (Welles 1938). It is also traceable in successive generations of optical, magnetic and digital sound technologies which offer increasingly higher signal-to-noise ratios, lower wow-and-flutter and harmonic distortion, higher dynamic range and wider frequency response. As a consequence, the migration from optical to magnetic to digital technologies has markedly assisted the general sophistication of sound design. Sonic information that was once crushingly limited to only a single monaural channel could became richer and denser as stereo, 5.1 or any of the other surround sound formats. To date, the most sophisticated soundfield technology is Dolby Atmos, a format which supports the processing of up to 128 discrete audio channels (plus associated metadata) and distributed to up to 64 speaker feeds (2012). Similarly, the impulse toward perceptual immersion is manifest in the development of production techniques such as the design of atmospheres, ADR and Foley. 152 Tan goes on to say the “distal cause of entertainment activity is an unconscious need for training useful capabilities, whereas the proximal cause is enjoyment of the activity for its own sake” (Tan, ESH 2008, p. 28). 153 Gallese has written further upon Feeling of Body and ‘liberated’ embodied simulation in relation to narrative and psychoanalysis, noting “the bodily affective self is at the roots of the narrative self” (2011b, p. 196). 154 The term narrative design is commonly encountered in the computergame industry where it stands in the stead of screenwriting. My use of the term here signals a desire to make commensurable the design processes of narrative and perceptual imagery. Narrative immersion is also referred to as transportation (Green & Brock 2000; Green & Donahue 2009; Holland 2008; Mar & Oatley 2008; Sestir & Green 2010; Tal-Or & Cohen 2010) or absorption (Bálint & Tan 2015). 155 The terms ‘bottom-up’ and ‘top-down’ processing are commonly used in cognitive neuroscience and cognitive psychology to refer to two distinct types of information processing. In the case of bottom-up processing, information sensed from the environment is processed in an emergent fashion to produce an ‘image’; that is, sensory information is processed ‘upwards’ to form a percept. In the case of top-down processing, internal models and expectations are processed ‘downwards’ through goal-directed attention. As described by Lawrence Barsalou:

At one extreme, bottom-up input activates sensory-motor representations in the absence of top-down processing (“pure” perception). At the other extreme, top-down processing activates sensory-motor representations in the absence of bottom-up processing (imagery and conception). In between lie processes that fuse complementary mixtures of bottom-up and top-down processing to coordinate the perception of physical entities (implicit memory, filling-in, anticipation, interpretation). (Barsalou 1999, p. 590)

For discussion of bottom-up and top-down processing in the contexts of animal and human cognition see Frans de Waal and Francesco Ferrari (de Waal & Ferrari 2010). For discussion of bottom-up and top-down processing in the contexts of emotion generation see Kevin Ochsner and associates (2009). 156 Both the Sucker Punch teaser and trailer are available on the accompanying DVD-ROM to Appendix D. 157 For a discussion of neural reuse theory see Michael Anderson (2010; 2014) and Yimeng Wang and John Bargh (2016). 158 In electing to use the term proto-narrative it may be I have inadvertently cast the balance too far in favouring a focus upon narrative. If proto-narrative is not, strictly speaking, narrative in nature why call this in any way ‘narrative’? My purpose is to indicate a structure, hitherto unremarked upon in Film Studies literature, which supports the emergence of narrative but is not narrative. Until another term usefully presents itself, proto-narrative will have to suffice. 159 For a discussion of the return of the cinema-of-attractions in post-classical cinema see Angela Ndalianis (2000) and the edited volume by Wanda Strauven (2006b). 160 Parallel to Wojciehowski & Gallese is Patrick Colm Hogan’s proposal for an affective narratology (2011). Hogan considers the structure and purpose of stories as inseparable from our emotion systems (2010, p. 65). 161 For example, Gerald Edelman’s contemplation in 1989 of the possibility of a biological theory of consciousness (Edelman 1989).

162 For a polar diagram which relates the spectrum of cognitivism, connectivism, and enactivism see Varela, Thompson & Rosch (1991, p. 7). 163 For a diagram describing a 360-degree spectrum of cognitivism, connectivism, and enactivism (divided into the quadrants of linguistics, philosophy, cognitive science and neuroscience) see Varela, Thompson & Rosch (1991, p. 7). 164 For a discussion of the problem of Cartesian anxiety with regard to developing a science of the mind see Varela, Thompson & Rosch (1991, pp. 133-145) and, more broadly, Overton (2002). 165 For a general overview of how emergence and enactivism stand in contrast to the computational approach of standard cognitive science see Shapiro (2011), particularly chapters 1 and 2. 166 For a detailed discussion of connectionism see Horgan (1996). 167 For a comprehensive overview of enactivism see Stewart, Gapenne & Di Paolo (2010). Precursors to enactivism may also be found in the work of Jean Piaget, Maurice Merleau-Ponty, and James J. Gibson. Beside Varela, Thompson and Rosch (Thompson, Evan 2007; Varela 1996; Varela & Depraz 2005; Varela, Thompson & Rosch 1991), other researchers of enactive cognition include Gerald Edelman (1992, 2004), Arthur Glenberg (1997, 1999), George Lakoff and Mark Johnson (Johnson, M 1987, 2007b; Lakoff 1987; Lakoff & Johnson 1980a, 1999), Andy Clark (1997, 1999, 2007), Kevin O'Regan (2001), Alva Noë (2004, 2009), and Shaun Gallagher (2005). 168 The interdisciplinary scope and ambition of the enactive approach is reflected in the traditions from which these authors herald: Varela is a biologist, Thompson a philosopher, and Rosch a psychologist. Indeed, among researchers of embodied cognition, there is a tangible spirit of interdisciplinary collaboration. For example, George Lakoff (linguistics) and Mark Johnson (philosophy) (1980a, 1999), J. Kevin O'Regan (psychology) and Alva Noë (philosophy) (O'Regan & Noë 2001), as well as Noë and Thompson (2002). Vittorio Gallese is particularly tireless in this regard, publishing with co-authors from a broad range of backgrounds: Vittorio Gallese (physiological and social neuroscience) and George Lakoff (linguistics) (2005), Gallese and David Freedberg (art history and psychology) (2007; 2007), Gallese and Corrado Sinigaglia (philosophy) (2011b), Gallese and Cinzia Di Dio (neuropsychology) (2012), Gallese and Hannah Wojciehowski (literary theory) (2011), and Gallese and Michele Guerra (film theory) (2012b, 2012a). 169 The concept of enaction spawned the term embodied cognition but there has been a recent movement toward the label ‘4E cognition’. This label usefully groups embodied cognition, embedded cognition, extended cognition, and enactive cognition. Richard Menary explains “[o]ne reason that the four E’s are grouped together is that they are all held to reject or at least radically reconfigure traditional cognitivism” (2010, p. 459). However, a conflation of these approaches as ‘4E cognition’ masks the differences in degree of their shared theoretical commitments particularly with regard to how ‘embodiment’ is defined (Martiny 2011, pp. 303-305). Related to the 4Es are the approaches of situated cognition and grounded cognition. Nevertheless, each of these terms foreground the notion of a body interacting with the environment within which it is embedded. 170 For further discussion of embodied cognition as a “research program within cognitive science” and not a unified theory in its own right see Shaun Gallagher (2015, pp. 96-97, emphasis in original). Theorists of embodied cognition exist along a continuum from mild to radical commitments regarding embodiment. Gallagher is at the radical end of this spectrum, and is critical of the ‘weak’ version of embodied cognition presented by Glenberg, Barsalou, Lakoff, Johnson, and Gallese. Embodied cognition is an exceedingly dynamic research programme and wrought with disagreements. For an insight into these internal disagreements, and the relevance of phenomenology to grounding the EC paradigm via its conceptualisation of embodiment, see Kristian Martiny (2011).

171 The claim that mediated environments are a virtualisation of real physical and social environments is not an idle one, and is developed in Chapter 5. See also Russell (2003, 2009) who proposes core affect may be altered as a consequence of the virtual reality experienced through cinema, theatre, literature and music. 172 See Margaret Wilson (2010) for a discussion of ‘cognitive technology’ and culture’s capacity to re-engineer the individual’s cognitive system. 173 I relate sensorimotor schema to image schema, and, like Johnson (1987), use the terms image schema and embodied schema interchangeably. 174 For a discussion of the legacy of Platonism in the study of cognition and emotion see Klaus Scherer (1995). Scherer argues the maintenance of a distinction between cognition and emotion is no longer useful. Of particular interest to Film Studies is Scherer’s observation that “Marx and Freud were unconscious Platonists” (1995, p. 1) in heeding a schism between cognition and emotion whereas Descartes and Spinoza initiated a movement toward integrating cognition and emotion. 175 For a discussion of the history of the pivotal role played by William James in reconfiguring the study of emotion see John Deigh (2014). Deigh observes that James’ approach “brought to completion the transformation of the study [of emotion] from one in moral philosophy to one in experimental psychology” (2014, p. 11) and contextualises James’ work with regard to British empiricism and, in particular, the work of David Hume. Deigh, however, is sceptical of both James’ argument and what he considers to be an incorrect interpretation of this work by neo-Jamesian theories of emotion. I offer Deigh’s critique only as an illustration of the ongoing disagreements within emotion science research. 176 It is worthy to note how the West in the 20th century became dominated by a general distrust of the emotions in favour of dispassionate and logical thought. This way of being in the world lingers still in contemporary society. It manifests, for example, along the fault lines which separate so-called ‘high’ from ‘low’ culture. For a discussion of the dubious assumptions concerning the relative value of art and entertainment within the ‘high’ and ‘low’ cultures of cinema see Lovell & Sergi (2009) and Bartsch (2016). 177 For a survey of emotion concepts, and the trends these concepts indicate, see Kleinginna & Kleinginna (1981) and Russell & Lemay (2004). For an overview of the abiding problems in defining the field of emotion research and producing

278 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. satisfactory definitions of emotion concepts see Frijda (2004). For a history of the development of the scientific study of emotion see Gendron & Barrett (2009). 178 For example, see accounts of the working definition of ‘emotion’ in Izard (2010) and Gendron (2010). 179 In claiming emotion to be a multicomponent process Russell turns to the Component Process Model developed by Klaus Scherer (2001; 1984a) and Clore & Ortony (2008). 180 See Russell’s virtual reality hypothesis (2003, pp. 155-156) for further discussion of the role of core affect in art and entertainment. 181 For an eloquent and concise history of the scientific study of emotion see Gendron & Barrett (2009). 182 Intriguingly, these higher levels of emotional communication can also place new constraints upon, and thus regulate, the lower levels (Bartsch & Hübner 2005, p. 5). This provides insight into how meta-emotions support moral reasoning, something Bartsch has subsequently explored in the context of cinematic media (Bartsch 2008, 2010, 2012; Bartsch & Hartmann 2017; Bartsch, Kalch & Oliver 2014; Bartsch & Oliver 2011). 183 It is worth noting at this point that the notion of ‘appraisal’ has drifted from the original definition developed by Magda Arnold. For a discussion of Arnold’s original concept of ‘direct appraisal’ and its slippage toward Lazarus’ concept of ‘cognitive appraisal’ see Arvid Kappas (2006). Kappas, moreover, argues that the definition adopted by Arnold and Zajonc are consistent with each other. 184 For a review of Zajonc’s work in emotion research and its historical significance see Berridge (2010), Niedenthal (2010), Niedenthal, Augustinova & Rychlowska (2010), and Winkielman (2010). 185 For examples, see Keith Oatley and Philip Johnson-Laird (1987), Andrew Ortony and Gerald Clore (Clore, Gerald L. & Ortony 2000; Clore, Gerald L. et al. 2005; Ortony, Clore & Collins 1988), and Nico Frijda (1984, 1986). 186 Such a cohort of researchers includes John Cacioppo (1998), John Kihlstrom (Kihlstrom 1984; Kihlstrom, Barnhardt & Tataryn 1992), John Bargh (Bargh 1994, 1997; Bargh & Chartrand 1999; Bargh & Morsella 2008), Paula Niedenthal (Niedenthal 1990, 2007; Niedenthal, Augustinova & Rychlowska 2010; Niedenthal & Maringer 2009), Kent Berridge (Berridge, K & Kringelbach 2008; Berridge, KC & Winkielman 2003), Piotr Winkielman (Winkielman & Berridge 2004; Winkielman, Niedenthal & Oberman 2008; Winkielman & Schooler 2008), Lindsay Oberman (Winkielman, Niedenthal & Oberman 2009), Lisa Feldman Barrett (Barrett, Ochsner & Gross 2007; Barrett & Wager 2006), and the neuroscientists Michael Gazzaniga (1998), Arne Öhman (Öhman 2002; Öhman, Flykt & Esteves 2001; Öhman, Flykt & Lundqvist 2000), António Damásio (1989b, 1989a, 1995, 1996, 2000, 2001), Joseph LeDoux (LeDoux 1991, 1994, 1995, 2000, 2007; LeDoux & Phelps 1993), and Vilayanur S. Ramachandran (Ramachandran & Hirstein 1999; Ramachandran & Seckel 2011). 187 Similarly, Arne Öhman has detailed the rapid, automatic and nonconscious orienting function of the fear network in the modality of vision (Öhman 2002; Öhman, Flykt & Esteves 2001), and his research indicates nonconscious emotional processing is robust across both natural and social environments, and the rapid nature of its activation suggests an evolved mechanism (Öhman et al. 2007, pp. 183-184; Öhman, Flykt & Lundqvist 2000). 188 Somatic markers are described by Bechara as bodily feelings, and the somatic marker hypothesis “posits that the overt or covert processing of somatic states provides the value mark [as either positive or negative] for a cognitive scenario” (Bechara, Antoine et al. 1994, p. 14, italics in original). 189 As a graphic example, Damásio elaborates upon this position through his examination of the famous case of Phineas Gage (Damásio, H et al. 1994). Extending Damásio’s work in somatic markers and decision-making is Michael Koenigs’ research into the role of the ventromedial prefrontal cortex, a neural region involved in emotion regulation and moral judgements (Koenigs et al. 2007; Young & Koenigs 2007), economic choice (Koenigs & Tranel 2007), as well as commercial brand preference (Koenigs & Tranel 2008). 190 For a detailed discussion of the nature of embodied emotion and its relationship to embodied cognition, as well as a comparison with amodal accounts of cognition, see Winkielman, Niedenthal & Oberman (2008). 191 For more on embodied emotions see Coutinho, Miranda & Cangelosi (2005), Halberstadt, Winkielman, Niedenthal & Dalle (2009), Prinz (2004), and Winkielman, McIntosh & Oberman (2009). 192 Emoacoustics is the portmanteau of emotional acoustics. It is concerned with the mechanisms by which everyday sounds induce affective reactions, and so casts a broader net than traditional theories of musical emotion. For a discussion of emoacoustics as the empirical study of psychoacoustical dimensions of emotional sound design see the work of Erkin Asutay (2012), Penny Bergman (2009; 2008), Ana Tajadura-Jiménez (2008; 2009; 2010; 2010), Aleksander Väljamäe (2007), and Daniel Västfjäll (2001, 2003; 2001; 2002). 193 The ‘science wars’ refers to a series of heated exchanges within the academy in the 1990s, and which appears to embody the spectre of C.P. Snow’s ‘two cultures’. The nature of the debate turns upon the differences between an account of human knowledge offered by scientific realism on the one hand and an account offered by postmodernism on the other. For examples of the position held by scientific realism see Gross & Levitt (1994) and Sokal & Bricmont (1998). For examples of the position held by postmodern cultural theorists see the edited volume Science wars (Ross 1996). 194 In parallel, Goldman & de Vignemont (2009) have proposed motoric, somato-sensory, affective and interoceptive bodily formats underpin symbolic representation. 195 As Edward Slingerland identifies it, “[t]he humanities–science divide, then, is fundamentally based upon mind–body dualism, and some – though by no means all – of the scholars who are eager to maintain a firewall between the two modes of inquiry are quite explicit about this” (2012, p. 76). 196 See Slavoj Žižek (2002) for an assessment of the strengths, weaknesses and antagonisms of consilience between Cultural Studies and the so-called ‘Third Culture’ of which E.O. Wilson is an exemplar. 197 One of the beneficial consequences of contemporary neuroscience research is the recent emergence of the field of cognitive cultural studies. Humanities scholars working in this nascent field include Amy Cook who applies theories of embodied cognition in the study of Shakespeare (2006, 2007, 2010), Patrick Colm Hogan who applies affective narratology in the study of literature and film (2003, 2008), Lisa Zunshine in the study of literature and television (2008a, 2008b, 2010),

279 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. and Rhonda Blair (2006, 2007, 2008, 2009a, 2009b) and Bruce McConachie (2001, 2008; 2006) in theatre and performance studies. The breadth of research undertaken by these scholars illustrates the degree to which vertical integration is already underway between the humanities and sciences.

199 For a review and critique of the reconfiguration of the relationship underway between cultural theory and science as a consequence of the ‘affective turn’ see Papoulias & Callard (2010) and Fitzgerald & Callard (2015). 200 For an overview of the history and key research goals of social cognitive neuroscience, a field which emerged only as recently as 2001, see Lieberman (2010). 201 Susan Hurley, critical of the classical ‘sandwich’ model of cognition, describes it thus:

A mainstream view of the mind has two components. The first is a view of perception and action as separate from each other and as peripheral. The second is a view of thought or cognition as the central core of the mind, at least for creatures with cognitive abilities. Cognition is virtually central, even if the mere implementation of cognitive processes is distributed. The mind decomposes vertically into modules: cognition interfaces between perception and action. Perception and action are not just separate from each other, but also separate from the higher processes of cognition. The mind is a kind of sandwich, and cognition is the filling. (Hurley 1998, pp. 401) 202 See, for example, studies concerned with imitation and learning (Iacoboni 2009a, 2009b; Iriki 2006; Rizzolatti, Giacomo, Fogassi & Gallese 2001), language evolution and acquisition (Cowley 2004; Gallese 2007a; Gazzola, Aziz-Zadeh & Keysers 2006; Rizzolatti, Giacomo & Arbib 1998; Tettamanti et al. 2005), action recognition, action understanding, and the understanding of another’s intentions (D’Ausilio 2007; Fogassi 2011; Fogassi et al. 2005; Galati et al. 2008; Iacoboni et al. 2005; Keysers et al. 2003; Kohler et al. 2002; Lahav, Saltzman & Schlaug 2007; Ocampo & Kritikos 2011; Petroni, Baguear & Della-Maggiore 2010; Umiltà et al. 2001), mind-reading of another’s emotional and mental state (Agnew, Bhakoo & Puri 2007; Rizzolatti, G. & Fabbri-Destro 2010; Rockwell 2008), Theory of Mind and social cognition (Gallese 2007a; Oberman & Ramachandran 2007; Ocampo & Kritikos 2011), intersubectivity (Bråten 2007; Decety & Lamm 2009; Pfeifer et al. 2008; Seligman 2009), and empathy (Bastiaansen, Thioux & Keysers 2009; Decety & Jackson 2004; Gallese 2001, 2003b; Gallese & Goldman 1998; Gazzola, Aziz-Zadeh & Keysers 2006; Iacoboni 2009a; Preston & de Waal 2003). 203 Intentionally referenced here is James J. Gibson’s concept of affordances (Gibson 1977). 204 Taking my cue from Mark Johnson (1987, p. 29), I use the terms embodied schema and image schema interchangeably. See Peter Gärdenfors (2007) for a discussion of the differences he perceives between Johnson’s use of embodied schema and image schema. See Tim Rohrer (2005) for a discussion of the relationship between image schema and sensorimotor schema. 205 Theodor Lipps developed the first scientific theory of aesthetic response to art (and, by extension, entertainment). This response he called Einfühlung (Montag, Gallinat & Heinz 2008). The concept later developed to include the empathic understanding of the minds of other beings. 206 A strong example of the action recognition and understanding function of audiovisual mirror neurons is the post- production sound effects technique of Foley. Foley is grounded in ‘in-character’ performance. For a concise history of Foley and a discussion of its aesthetic as a performance see Wright (2014). 207 According to Mark Johnson image schemas

… are recurring patterns of organism-environment interactions that exist in the felt qualities of our experience, understanding, and thought. Image schemas are the sort of structures that demarcate the basic contours of our experience as embodied creatures. They depend on how our brains work, what our physiology is like, and the kinds of environments we inhabit. They are one of the most basic means we have for discrimination, differentiation, and determination within our experience. Their philosophical significance, in other words, lies in the way they bind together body and mind, inner and outer, and thought and feeling. They are an essential part of embodied meaning and provide the basis for much of our abstract inference. (Johnson, M 2005, p. 31, emphasis in original)

However, Gallese & Lakoff hesitate

… to call schemas “concepts,” simply because concepts have long been traditionally thought of as being direct reflections or representations of external reality. Schemas are clearly not that at all. Schemas are interactional, arising from (1) the nature of our bodies, (2) the nature of our brains, and (3) the nature of our social and physical interactions in the world. Schemas are therefore not purely internal, nor are they purely representations of external reality. We will, for the moment, think of concepts as schemas, though that idea will be extended below when we discuss abstractions. (Gallese & Lakoff 2005, p. 467) 208 For an example of audiovisual metaphor, see Chris Cunningham’s TVC Engine (1999) available on the accompanying DVD-ROM to Appendix D. 209 Audiovisual fusion refers to the mandatory integration of information from several input modalities. For aural and visual information this integration occurs when presented within a temporal window of –30 ms to +170 ms (van Wassenhove, Grant & Poeppel 2007, p. 598). 210 It should be noted, however, that Gallese & Guerra speak only to visual imagery when applying ES theory to cinema thus leaving undiscussed the function of auditory imagery and sound design. 211 Gallese has written further upon Feeling of Body and liberated embodied simulation in relation to narrative and psychoanalysis, noting that “the bodily affective self is at the roots of the narrative self” (2011b, p. 196). 212 Sensory information is autonomically parsed for affective content. Gilles Pourtois, Beatrice de Gelder and Jean Vroomen claim, “binding between affective information that is seen as well as heard triggers a mandatory process of audiovisual

280 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. integration” (2000, p .1329). Crossmodal binding “does not await the outcome of separate modality specific decisions and is not under attentional control” (Pourtois et al. 2000, p .1329). 213 For an example of cinematic beat, entrainment and ‘chills’ see the theatrical trailer for We Are Your Friends (Joseph 2015b) provided on the DVD-ROM accompanying Appendix D. The trailer exemplifies rhythmic entrainment twofold as it illustrates entrainment of the party crowd within the diegesis even as it acts to entrain the audience of the trailer. 214 This sentiment is echoed by Gad Saad when he observes that “rather than viewing the human mind as a blank slate that is subsequently shaped by arbitrary (if not patriarchal and sexist) products of popular culture, I propose that many aspects of popular culture are invariant to time and place precisely because they are indicative of our evolved human nature” (2012, p. 109). 215 “[H]uman biology makes culture possible, whereas culture makes the biological survival of humans possible” (Gerstein, Rountree & Ordonez 2007, p. 378). Michael Wheeler and Andy Clark (2008) call the matrix of interaction between genes, embodiment and culture the “triple helix”, and the recently emerged interdisciplinary field of cultural neuroscience (Chiao 2009a; Duque et al. 2010; Han & Northoff 2008, 2009; Li 2003; Vogeley & Roepstorff 2009) sets itself the task of describing the co-defining nature of human biology and culture. 216 There is a dearth of research which specifically examines the relationship between environmental sound and affect. The current research field of emoacoustics, a portmanteau of ‘emotional acoustics’, responds to this lack. Emoacoustics research is represented by the work of Ana Tajadura-Jiménez, Aleksander Väljamäe, Erkin Asutay, and Daniel Västfjäll (Asutay et al. 2012; Tajadura-Jiménez 2008; Tajadura-Jiménez et al. 2009; Tajadura-JiménezLarsson, et al. 2010; Tajadura- JiménezVäljamäe, et al. 2010; Väljamäe & Tajadura-Jiménez 2007). 217 Referencing Peter Gärdenfors (2003), Juslin et. al. argue that “one may think of these mechanisms as based on a number of distinct ‘brain functions’, which have developed gradually and in a particular order during the evolutionary process, from simple sensations to syntactical processing” (Juslin et al. 2010, p.620). The acronym BRECVEM reflects these mechanisms in order of evolutionary development (Juslin & Västfjäll 2008, p. 568). 218 Indeed, as Juslin, Liljeström, Västfjäll & Lundqvist (2010, p. 616) report, cognitive appraisal is the least significant factor in the musical induction of emotion. So much so, it was omitted from the BRECVEMA framework. 219 As Smith admits, his proposal for filmic mood-cue “may also apply to other audiovisual narrative media that unfold uninterrupted in real time (for instance, certain kinds of theater)” (Smith, GM 2003b, see note 5, p. 195). 220 It is interesting to note, however, that Bartsch takes up Smith’s concept of mood-cue when developing her theory of social-moral meaning based in embodied emotion metaphors (Bartsch 2016, pp. 206-207). 221 I omit Grodal here because his PECMA flow model (Grodal 2006, 2009a, 2009c) exhibits a strong bottom-up construction of narrative and so appears to wedge between the positions of Carroll and Tan on the one hand and Smith on the other. The PECMA flow model is similar to Smith’s mood-cue approach except that Grodal tends to focus upon narrative and genre as cinema’s natural endpoints (as per Carroll and Tan). Smith’s mood-cue approach is essentially non-narrative in nature but may be recruited to generate narrative. In addition, Smith’s mood-cue approach is similar to Grodal’s PECMA flow model in that it draws upon a wide variety of stylistic cues to structure affect locally and globally, but is not constrained to the level of character motivation. 222 Although Smith tends to refer to the visual, he acknowledges the multimodality of cinematic imagery. 223 However, Smith lacks detail in specifying the underlying mechanisms of visual induction and the possible nature of crossmodal interaction. 224 Meta-emotions are “symbolic meaning structures” (Bartsch 2008, p. 48) which evaluate “thoughts and feelings about emotions that are represented and/or elicited in the viewer” (Bartsch 2008, p. 45). Meta-emotions may be cued by the “aesthetic representation of emotions; the narrative context of emotions; and symbolic elements that refer to cultural norms and values concerning emotions” (Bartsch 2008, p. 45). 225 The explanatory gap (Levine 1983; Nagel, T 1974) refers to the failure to explain how the experience of qualia and consciousness arise from a physical basis. At the root of the explanatory gap is the cognitive inaccessibility of phenomenal experience. See Block (2009, pp. 1113-1115) for a detailed description of the explanatory gap and its relationship to dualism. 226 Block considers this approach to date from at least Democritus (c. 460–370 BCE) and notes its reemergence in the modern literature with U.T. Place’s ‘Is consciousness a brain process?’ (1956), J.C.C. Smart’s ‘Sensations and brain processes’ (1959), and Herbert Feigl’s ‘The “mental” and the “physical”’ (1958). Early developers of contemporary Biological Framework theory are Gerald Edelman (Edelman 1987, 1989, 1992, 2003, 2004; Edelman & Mountcastle 1978; Edelman & Tononi 2001) and Francis Crick and Christof Koch (Crick 1994; Crick & Koch 1990, 1998). 227 Block's conclusion is, of course, contentious and debate continues within the field. Recent findings indicate both GWT and HOT theories continue to offer valuable insights (Baars 2003; Lau & Rosenthal 2011) even as HOT theory is claimed to be defunct (Block 2011a; Sebastián 2014). 228 In contrast, William Lycan (1996, pp. 2-4) proposes at least eight forms of consciousness: i) organism consciousness, ii) control consciousness, iii) consciousness-of, iv) state/event consciousness, v) reportability, vi) introspective consciousness, vii) subjective consciousness, and viii) self-consciousness. However, as Lycan himself notes, some of these types are similar to, or overlap, those proposed by Block. An instance of this is Lycan’s association of control consciousness with Block’s concept of access consciousness (Lycan 1996, pp. 2-3). 229 Here Block borrows Thomas Nagel’s famous expression which describes phenomenal experience as “what it is like” (Nagel, T 1974). 230 However, as with much in Consciousness Studies, debate is fierce. For arguments against Block’s distinction between P- and A-consciousness see Schlicht (2012). Similarly, cognitive overflow is also a contested concept. For arguments against Block’s concept of cognitive overflow see Brown (2012). For a discussion of the concept’s strengths and weaknesses see the multiple responses to Block’s target article (2007b); for example, Koch & Naotsugu (Koch & Tsuchiya 2007) who are in support, and Sergent & Rees (2007) who are not.

231 To recap how the powerfully evolved function of nonconscious affect shapes and guides conscious thought see Arne Öhman (Öhman 2002; Öhman et al. 2007; Öhman, Flykt & Lundqvist 2000; Öhman & Wiens 2003), Piotr Winkielman and Kent Berridge (Berridge, KC & Winkielman 2003; Winkielman 2010; Winkielman & Berridge 2004; Winkielman, Berridge & Wilbarger 2005; Winkielman & Schooler 2011), Regina Lapate et. al. (2014), and Beatrice de Gelder (de Gelder 2005; de Gelder & Tamietto 2011; de Gelder, van Honk & Tamietto 2011). 232 As Bartsch and Hübner contend:

The emergence of superordinate skills leads to a reorganization and differentiation of the basic skills. Nevertheless, the skills on the subordinate levels continue to function independently of the newly acquired level of cognitive complexity. Thus the subordinate levels can work without the superordinate levels, but not vice versa. (Bartsch 2004, p.5) 233 Damásio defines an ‘image’ as “a mental pattern in any of the sense modalities” but also includes other forms of imagery such as interoception, the image of inner bodily awareness (Damásio, AR 1999, pp. 8-9). 234 In the quote below, the Two Williams Debate refers to the opposing positions taken by Wilhelm Wundt and William James regarding the basis of human knowledge. Wundt posited knowledge as a priori and efferent to the human sensory array whereas James argued knowledge as a posteriori and afferent. 235 Such ‘situatedness’ suggests similarities with James J. Gibson’s concept of affordance (Gibson 1977) and Barsalou’s theory of ‘situated’ cognition (Barsalou 2003a, 2009; Yeh & Barsalou 2006). 236 For a discussion of the automaticity of human behaviour see Marc Jeannerod (Fourneret & Jeannerod 1998; Jeannerod 2006), John Bargh and Tanya Chartrand (Bargh 1994, 2011; Bargh & Chartrand 1999; Chartrand, van Baaren & Bargh 2006), and Daniel Wegner (2002, 2003; 1998). For a discussion of Unconscious thought theory see Ap Dijksterhuis and Loran Nordgren (2006). 237 In particular, see Daniel Wegner’s The illusion of conscious will (2002) for an exploration of the phenomenon. 238 According to Susan Hurley: “A mainstream view of the mind has two components. The first is a view of perception and action as separate from each other and as peripheral. The second is a view of thought or cognition as the central core of the mind, at least for creatures with cognitive abilities. Cognition is virtually central, even if the mere implementation of cognitive processes is distributed. The mind decomposes vertically into modules: cognition interfaces between perception and action. Perception and action are not just separate from each other, but also separate from the higher processes of cognition. The mind is a kind of sandwich, and cognition is the filling.” (Hurley 1998, pp. 401) 239 For examples of this argument see John Bargh and Tanya Chartrand (1994, 2011; 1999) and Susan Pockett (2004). 240 For a discussion of the biological basis of social cognition and interaction see Frith and colleagues (Frith, CD 2007; Frith, CD & Frith 1999, 2005, 2007, 2012; Frith, U & Frith 2001, 2010). 241 Indeed, narrative and the making of music appear to be true cross-cultural universals. For a discussion of the universality of music see Mithen (2006). 242 Consciousness, or, more correctly, that part of consciousness we call conscious thought, is a comparatively recent evolutionary development. For a discussion of the antiquity of the cognitive unconscious and the subsequent evolutionary development of consciousness see Rolf Reber (1989, 1992) and Bernard Baars (1992). 243 The term “bodymind” was coined by John Dewey and is used extensively by Mark Johnson. In particular, see Johnson (2007b, pp. 7-15). 244 For further reading on gene-culture co-evolution, Dual Inheritance Theory and bioculturalism see Lumsden & Wilson (1981) Cavalli-Sforza & Feldman (1981), and Boyd & Richerson (Boyd & Richerson 1985, 2005; Richerson, Peter J. & Boyd 2000; Richerson, Peter J. & Boyd 2001; Richerson, Peter J. & Boyd 2005a; Richerson, Peter J., Boyd & Henrich 2010). 245 In this thesis, computergames are considered to be a form of cinematic media, where the minimum requirement is the dynamic manipulation of affective audiovisual imagery. 246 Consider, for example, Léger & Murphy's avant-garde Ballet Mechanique (1924) or the music videos for The Chemical Brother’s Star Guitar (Gondry 2001b), Autechre’s Second Bad Vilbel (Cunningham 1996) and Hot Chip’s Huarache Lights (Knowles 2015). Star Guitar, Second Bad Vilbel, and Huarache Lights are available on the accompanying DVD-ROM to Appendix D. 247 For more on the ‘turn’ to narrative, see Martin Kreiswirth (1992, 1995, 2005) and Matti Hyvärinen (2008, 2010, 2012). 248 Even within this polarised environment, it is interesting to note the simultaneous release of ‘high concept’ movies (such as Gravity (Cuarón 2013b) and Interstellar (Nolan 2014)) across both standard multiplex and large-format (e.g. IMAX) auditoria. 249 Examples of such networks include HBO, Showtime, Netflix, and Amazon Prime. 250 Narrative immersion is also referred to as transportation (Green & Brock 2000; Green & Donahue 2009; Holland 2008; Mar & Oatley 2008; Sestir & Green 2010; Tal-Or & Cohen 2010) or absorption (Bálint & Tan 2015). 251 The term narrative design is commonly encountered in the computergame industry where it stands in the stead of screenwriter or author. My use of the term signals a desire to make commensurable the design processes of narrative and perceptual imagery. 252 Gallese has written further upon Feeling of Body and ‘liberated’ embodied simulation in relation to narrative and psychoanalysis, noting “the bodily affective self is at the roots of the narrative self” (Gallese 2011b, p. 196). 253 Charlotte Jensen notes, in the case of movie trailers, “narration is reduced for the benefit of intensified emotion” (2014, p. 123). Jensen’s observation implies interacting mechanisms for the reduction or intensification of narrative and emotion, but fails to identify a corollary of such narrative design. The design of perceptual qualities remains the untheorised partner. This blindspot arises, in part, due to Jensen’s reliance upon the cognitive film theory of Bordwell and Grodal which privileges understanding of the mental processes which activate narrative schemata. However, it should be said that Grodal’s approach

282 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. is less ‘cognitive’ and more ‘embodied’ than that of Bordwell. Grodal’s PECMA flow model (Grodal 2009c) describes an embodied mechanism but it does, nevertheless, have a gravitational tendency toward privileging narrative. 254 Specifically, these music cues are Immediate Music’s Prologue to a Quest (2009) and Lords of Acid’s The Crablouse (1994). The teaser soundtrack also features some of the movie’s diegetic dialogue (used as voice-over) as well as sound effects (used quasi-diegetically). 255 For a cinematic example of beat, synchronisation and entrainment see We Are Your Friends (Joseph 2015b). A trailer for the movie (available on the accompanying DVD-ROM to Appendix D) exemplifies this kind of entrainment, both as it occurs for the party crowd within the diegesis and for the audience of the movie. 256 See, for example, Grahn, Henry & McAuley who found audition primes visual beat perception but not vice versa, and that “hearing an auditory rhythm creates an internal representation that encourages an auditory recoding of a subsequently presented visual rhythm” (Grahn, Henry & McAuley 2011, p. 1241). 257 Although I have attempted here to describe the flow of the sequence, it is important to note that the actual temporal structure of the audiovisual elements is all but lost and, moreover, does not elicit the effect of aesthetic chills. Please audition the Sucker Punch teaser, available on the accompanying DVD-ROM to Appendix D, to experience the effect. 258 The free-floating nature of core affect has special significance for cinematic media because “[c]ore affect is altered by real events, but also by imaginary, remembered, and foretold events; core affect responds to virtual reality in art, imagination, fantasy, and entertainment” (Russell 2009, p. 1266). See Russell’s virtual reality hypothesis (Russell 2003, pp. 155-156) for further discussion of the role of core affect in art and entertainment. 259 Narrative takes time to create. It is interesting to note the teaser is 1:30 min:sec in duration and requires significantly less time to achieve its effect than the trailer (2:30 min:sec). 260 At only 60 seconds in duration, Spike Jonze’ TVC Lamp (2002b) provides another compact example of cinematic narrative design. Extraordinarily, a theory of mind and the attribution of character is projected onto an inanimate object, the lamp itself. Lamp is provided on the DVD-ROM accompanying Appendix D. 261 See Edward Branigan (1992b) for a discussion of narrative schema. See William Brewer (1981) for a discussion of event schema. For a discussion of event perception, boundaries and segmentation from the perspective of cognitive psychology see Jeffrey M. Zacks and associates (Zacks et al. 2001; Zacks et al. 2007; Zacks & Swallow 2007; Zacks & Tversky 2001), and, particularly relevant to theories of event segmentation and filmic narration, (Magliano & Zacks 2011; Zacks & Magliano 2011; Zacks et al. 2010). 262 In this regard I follow Mark Johnson who, in turn, follows John Dewey in recognising the intrinsic unity of the human body–mind (Johnson, M 2007b, pp. 7 and 11). 263 Marco Caracciolo (2012, 2014), for one, has taken up this challenge. 264 For a discussion of neural reuse theory see Michael Anderson (2010; 2014) and Yimeng Wang and John A. Bargh (2016). 265 For an broad exploration of audiovisual synchronisation and fusion, or what Chion calls synchresis, see K.J. Donnelly’s Occult aesthetics: synchronization in sound film (2014). 266 André Gaudreault and Tom Gunning are accredited with development of the concept of “cinema of attraction” in a 1986 joint paper entitled ‘Le cinéma des premier temps: un défi a histoire du film?' (Gaudreault & Gunning 1989). However, the term was neither explicitly named nor defined until Gunning’s 'The cinema of attraction: early film, its spectator and the Avant-Garde' (1986) and further refined as ‘cinema of attractions’ in 'The cinema of attractions: early film, its spectator and the Avant-Garde' (Gunning 1990). Wanda Strauven (2006a) has traced the concept’s formation, maturation and re- deployment in contemporary cinema. 267 Debate surrounds the existence of a clear differentiation between kinaesthesia and proprioception and, as a result, these terms are often used interchangeably. However, kinaesthesia is concerned with the ‘feeling of movement’, while proprioception provides feedback that the body has moved. The rollercoaster-as-event exploits a tension between kinaesthetic and proprioceptive systems; the body is felt to be moving in a way that is inconsistent with the felt status of the body’s internal conditions. We do not feel our body’s muscles, tendons or internal organs in the act of moving, and yet we move. As a passenger riding a rollercoaster, there is a radical mismatch between the intense feeling of motion and the imperfect feedback of having exerted the prerequisite effort to achieve such motion. Of course, this is exactly the attraction of rollercoasters: the playful, if extreme, manipulation of a felt experience of gravity. This playful manipulation can itself be considered a form of experimentation and learning. 268 The definition of an emotional roller coaster (Thatcher 2009) is available on the accompanying DVD-ROM to Appendix D. 269 Vines, Nuzzo & Levitin (2005, p. 139) consider ‘tension’ to correspond to the dimension of arousal more so than valency. In analysing the affective nature of a rollercoaster, I note the experience to possess high arousal but also a dimension of valency which may be either negative (fear and terror) or positive (exhilaration and joy). For a discussion of the relationship between tension, arousal and valency in the context of ‘musical tension’ see Lerdahl & Krumhansl (2007), Krumhansl & Lerdahl (2011), and Vines, Nuzzo & Levitin (2005). 270 At this point, such is the terror which grips him, the boy is rendered incapable of speech and appears to exhibit mammalian freezing behaviour. 271 Between 01:35–01:42 min:sec, emotion-markers 9–12 are a minor reprise of the apprehension–fear–coping–joy sequence. 272 I might go so far as to call this structure an emotion episode. Media psychologist Ed Tan cites Frijda, Mesquita, Sonnemans & van Goozen (1991) in defining an emotion episode as “a continuous emotion sequence resulting from the more or less continuous impact of one given event or series of events” (Tan, ESH 1996, p. 58). Tan deploys the term at the macro level of narrative whereas I focus at a non-narrative level of emotional structure. 273 Façade is a language-driven narrative which deploys low-fidelity cinematic imagery. For more information of Façade see Interactivestory.net (Mateas & Stern 2005a).

274 See also Kristen A. Lindquist (2013) for a detailed argument in support of the psychological constructionist approach to emotion categories. Lindquist not only claims “emotions are mental events that are constructed of more basic psychological processes” (2013, p. 356) but “sensations from the body [core affect] form a basic psychological ingredient that contributes to all mental states” (2013, p. 364). 275 Schell worked for Disney Imagineering between 1995 and 2002, and is currently a professor in computer science and entertainment technology at Carnegie Mellon University. The design of a rollercoaster is the design of an experience which consistently and reliably produces a satisfying episode of emotions, and it is this emotion episode which is the object of commercial exchange. Schell notes “the quality of an entertainment experience can be measured by the extent to which its unfolding sequence of events is able to hold a guest’s interest” (Schell 2008b, p. 247). 276 Freytag identifies five phases which comprise the familiar classical arc of Aristotelian dramatic structure: exposition, rising action, climax, falling action, and dénouement (Prince, G 2003, p. 1980). Allen Tilley finds Freytag’s Pyramid to be something of a “blunt instrument” that “does not answer very well to anything more complex than a brief anecdote” and somewhat dismissively considers it only an “episode” or “partial narrative” (Tilley 1992, pp. 3-4). For the purposes of literary criticism and analysis I do not doubt this to be the case. My point here is that proto-narrative is an atom from which narrative proper is constructed, and may be undervalued by literary theorists who are focused on narrative superstructure and not affective substructure. 277 It is tempting to draw an analogy between Vines, Nuzzo & Levitin’s notion of musical tension as an accumulation and discharge of affective energy within the listener and the Aristotelian concept of catharsis. However, I resist the temptation. Catharsis is often presented as a dramaturgical fact, but social and cognitive psychologists have found little evidence for the existence of catharsis as a genuine phenomenon (Bushman 2002; Geen & Quanty 1977; Zillmann 2000). A more convincing theory is to be found in Dolf Zillmann’s Excitation Transfer theory (Bryant & Miron 2003; Zillmann 1971, 1983). Currently, however, there is an attempt to salvage and rehabilitate catharsis as a useful concept (Khoo & Oliver 2013; Scheele 2001; Scheele & DuBois 2006). My point here is that proto-narrative is a non-narrative affective structure co-opted by narrative. 278 For a discussion of tension, temporal expectancy and filmic narrative see Minet de Wied (1995). 279 The ‘Tokyo Tonight’ sequence occurs at approximately 1:02:30–1:12:20 (hh:mm:ss), and is available on the accompanying DVD-ROM to Appendix D. 280 See Geoff King (2000) for a comprehensive discussion of the Hollywood blockbuster. 281 The T-rex battle sequence performs a crucial and empathic function within the movie. Throughout the sequence Kong persistently places himself between Darrow and physical jeopardy and his actions take on a moral dimension: a decision to protect Darrow from the dangers of his world. The audiovisual design of the sequence viscerally immerses the audience into a situation of sacrifice, and this perceptual immersion is vital for the communication of the overarching themes of the narrative. Toward the end of the movie, when the situation is reversed, Darrow is unable to protect Kong from the dangers of her world and, as a consequence of her thwarted attempts to fulfil her moral responsibility, the audience experiences tragedy at Kong and Darrow’s defeat. 282 The ‘T-rex battle’ sequence occurs at approximately 1:36:00–1:47:00 (hh:mm:ss), and is available on the accompanying DVD-ROM to Appendix D. 283 From a character perspective, the sequence focalises almost exclusively on Darrow, switches to Kong, switches to include both Kong and Darrow, until finally switching back to concentrate on Darrow. Auditory space rapidly expands and contracts – from extreme close perspective (such as shifts in breathing rhythms) to extreme wide perspective (such as Darrow’s blood- curdling scream as she falls away into the crevasse) – as threat increases and decreases. 284 It is interesting to note the effect of such cinematic dynamism on the comprehension of character. The Ann Darrow of the 1933 version is emotionally immobilised by fear, visually immobilised by the frame, and physically immobilised by a tree; this Darrow is in desperate need of saving. The Ann Darrow of the 2005 version is always on the move, and a resourceful agent for her own survival even if the odds are against her. 285 Star Guitar (Gondry 2001b) is available on the accompanying DVD-ROM to Appendix D. 286 For an insight into Gondry’s creative process whereby musical objects become associated with and mapped as visual objects see Making of Star Guitar (Gondry 2001a) which is available on the accompanying DVD-ROM to Appendix D. 287 In this way Star Guitar resembles its forebear Steamboat Willie (Iwerks 1928). Steamboat was a radical departure from the production techniques of the time because it animated vision to a completed musical soundtrack. The effect was a soundtrack that appeared to drive visual imagery: sound brings forth the world. 288 Minogue is layered and positioned onscreen to create visual interpretations of the aural layering (i.e. multitracking, harmonising and chorus effects) of the music track. 289 Minogue is both apart from and a part of the mediated world, and knowingly breaks the fourth wall to directly address the audience. 290 Where Star Guitar is sparse, linear and modestly complicating and Come into My World is circular and rapidly cumulative in its complication, Around the World is dominated by complex multidirectional lines of force moving in XYZ space. 291 The music videos Star Guitar (Gondry 2001b), Come into My World (Gondry 2002) and Around the World (Gondry 1997) are available on the accompanying DVD-ROM to Appendix D. 292 Although the term describes a cinematic unit, proto-narrative is itself comprised of affective beats. These beats are allied to what Greg M. Smith calls emotion cues and what Tom Gunning calls a “burst of presence” (Gunning 1993, p. 6). 293 For an evolutionary psychological approach to the question see Katja Mellmann (2011, 2012), and for an evolutionary biological approach see Karl Eibl (2012). 294 Mellmann acknowledges narrative may be a “contingent universal, that is, a cultural behavior which notably relies on biological substrates and therefore emerges in similar fashions in all human cultures, times, and milieus” (Mellmann 2012, p. 30). For a discussion of the evolutionary development of mimesis, diegesis and narrative from the perspective of literary theory see H. Porter Abbott (2000, 2003).

295 In ways similar to Baumeister (2010), Sugiyama considers the social environment to be equivalent with physical environment from an evolutionary perspective:

[b]ecause a major selection pressure on humans has been humans themselves, ancestral humans needed to construct a map of their social world. The ability to attribute mental states to others is necessary for this map, but not sufficient: a social map must show the intentions, emotions and beliefs of individuals relative to one another. (Scalise Sugiyama 2009, p. 94) 296 Similarly, the T-rex battle in King Kong (2005) provides the same opportunities for emotional regulation and learning but with the additional dimension of being embedded into a social and moral context. 297 But note Altman was experimenting with multi-threaded narrative as early as Nashville (1975). 298 Australia has the somewhat dubious accolade of being the world's biggest illegal downloader of Game of Thrones (Game of Thrones 2011– ) (Collins 2014; Rothman 2014). To circumvent HBO’s staggered global release dates, Australian audiences actively seek out services that remove geo-blocking or log on to torrent sites which provide pirate access. 299 Ohler & Nieding propose play to be the “phylogenetic cornerstone for modem media entertainment” (2006b, p. 429). 300 For a discussion of the constraints placed upon human social network size by neuroanatomy (e.g. neocortex and amygdala volume) see (Bickart et al. 2011; Dunbar 1992, 1993, 1998). For a discussion on the effects of limited working memory on cognition, behaviour and culture see Torkel Klingberg’s The overflowing brain: information overload and the limits of working memory (2009) and Nelson Cowan’s ‘The magical mystery four: how is working memory capacity limited, and why?’ (2010). 301 The notion that pop culture plays an important and positive role in expanding human cognitive capacities is taken up by Steven Johnson who suggests “popular culture is actually making us smarter” (2005a, 2005b). Johnson advances this argument using the examples of multi-threaded narrative television drama, in particular The Sopranos (The Sopranos 1999– 2007) and 24 (24 2001–2010). 302 For more on Gumbrecht’s concept of a productive tension in the oscillation between presence and meaning see (Gumbrecht 2004, p. 106 and 116). 303 One need only think of the many, many neologisms coined by Chion (2012). Oftentimes, Chion’s neologisms merely label filmic phenomena rather than provide deeper explanations. Some of these neologisms – synchresis, for example – is the re- labelling of phenomenon more fully explained in the perceptual sciences as crossmodal fusion. 304 Such a definition of cinematic imagery clears the way for future technologies to incorporate other modalities such as touch, taste, olfaction and proprioception. While the molecular nature of scent has proven to be a technological stumbling block for its ready mediation, this may not prove to be insurmountable for touch. The laboratories of Disney Research are prototyping tactile technologies, most notably REVEL. Olivier Bau and Ivan Poupyrev (2012; 2012; 2010) describe REVEL as a tactile feedback technology for augmented reality. The addition of haptic to aural and visual immersion in virtual environments, of course, expands further the sphere of the cinematic. 305 For a general discussion of ‘narrative emotions’ see Suzanne Keen (2011). Note that Keen (Keen 2015) regards curiosity, surprise and suspense as ‘narrative emotions’ although surprise is also the hallmark of cinema-of-attractions, and also considered to be one of the so-called ‘basic emotions’. 306 It is apposite both Robert Zajonc (1980, p. 151) and Mark Johnson (2007b, p. 52) reference e.e. cummings’ poem since feeling is first (1926) as a touchstone to affirm “[b]efore there is abstract thinking, before there is reasoning, before there is speech, there is emotion” (Johnson, M 2007b, p. 52). 307 Contra Pylyshyn argues mental imagery are language-like symbolic representations. 308 Auguste Rodin, in critiquing the static instant of the photographic image, claims sculpture to more truthfully depict movement precisely because of the artform's capacity to compress time and communicate movement (Halloran 2007, p. 219). In this way, Rodin suggests how sculpture may be said to possess ‘cinematic’ qualities albeit in weak form. 309 The sound design process for the creation of sound effects (SFX) and atmospheres of In the Cut is somewhat unconventional. Ordinarily, a single individual (or small group) is responsible for either SFX or atmospheres across a movie. In the case of In the Cut, Miller and I divided the movie according to scene location so that we were each individually responsible for both the SFX and atmospheres of specific environments. This allowed for an intimate evolution of each locale’s environmental soundscape through which we shaped an emotional landscape. For a discussion of the design strategy employed for voice see Helen Macallan and Andrew Plain (2010, pp. 253-255). Plain is the supervising sound editor of In the Cut. 310 The Richard King interview occurs in The sound on Inception (Coleman 2011), and is available on the accompanying DVD-ROM to Appendix D. 311 Walter Murch (2017) describes the distinction between sound editing and sound mixing as the distinction between strategy and tactics respectively, to which I would add that sound mixing is the management of audience attention and the emulation of point-of-audition. 312 It should be acknowledged that sound effects and their temporally extended cousins – atmospheres – are aspects of environmental sound and differ only in their foreground/background relationship. When an element of an atmosphere is foregrounded it becomes a sound effect but when a sound effect recedes to the background it is subsumed into atmosphere. 313 In the Cut (Campion 2003) is available in its entirety on the accompanying DVD-ROM to Appendix D. 314 This dovetailing of sound design and music score is worthy of note if only because it arose spontaneously and unbidden. Created separately (the sound design in Sydney and the music score in Reykjavík), with no contact between the two teams, the sound design and music, remarkably, exhibit similar timbral profiles. Atmospheres, sound effects and music morph from one into the other without any clear demarcation. 315 The sync sound of the original material suggested only a slight movement through trees but this was enough for me to anchor and amplify the construction of something much more significant.

316 See Tajadura-Jiménez and colleagues (Tajadura-JiménezLarsson, et al. 2010) for an examination of the affective response to spatial acoustics. 317 When experiencing an aesthetic emotion such as the sublime it is interesting to note the correlation of fear and awe. See Konečni (2005) for a discussion of the sublime, and the evolutionary function of aesthetic emotions. 318 However, there are signs this may be changing. In 2017, Arrival (Villeneuve 2016) won the Oscar for Best Sound Editing, and although this movie is from the traditionally ‘noisy’ genre of science fiction, the sound design is remarkably intimate and restrained. 319 One such is an IMDb user who observed: “Anyone who appreciates film sound should watch (or rather, listen to) ‘In the Cut’ because it's one of the few existing films that uses 5.1-channel sound for more than SFX gimmicks or making sure the Dolby Digital logo appears on its DVD case. The film creates real ambiance and mood with its sound mix, which helped suck me into the story world and get a sense of the characters' environment. I first noticed this when Frannie descends the stairs in the restaurant (just before she sees the mysterious villain). As she walks through the noisy crowd and down the stairwell, the conversations, bustling and other background fade from the front to rear channels and mix with her footsteps as she descends. This, to me, is much more elegant use of 5.1-channel surround than sticking a few whizzing noises in the rear channels when a spaceship flies off the top edge of the frame. ‘In the Cut’ makes full use of its available channels, which is more than 99% of high-budget films can say.” (squirrelsatemynuts 2005) 320 In the Cut represents a departure from the high literary mode of The Piano (Campion 1993) and The Portrait of a Lady (Campion 1996) which established Campion as an auteur director, and to which she subsequently returned with Bright Star (Campion 2009). Even An Angel at My Table (Campion 1990), a television series also released theatrically, exhibits an early fascination with literature and the lives of authors, in this case that of Janet Frame. For more on Campion’s cinematic oeuvre and literary adaptation see Gelder (1999). 321 Edward Hall (1963, 1966) termed the study of the segmentation of human space as proxemics. These spatial zones exist pan-culturally, but are modulated by cultural rules. In this way, proxemics can be understood as both a biological–ecological understanding of inhabited space as well as providing basis for a study of social semiotics. The proxemics of In the Cut arises from the cinematic manipulation of peripersonal space. 322 Specifically, consider the sequences at approximately 00:11:10–00:14:05, 01:15:53–01:17:50, and 01:19:25–01:23:02 (hh:mm:ss). 323 An impressive cinematic example of this kind of affective dilation of time is the bullet-time sequence from The Matrix (Wachowski & Wachowski 1999). Although opportunity does not permit a thorough examination of the impact of affect upon subjective temporality, several significant studies should be mentioned in passing, in particular (Bar-Haim et al. 2010; Droit-Volet, Fayolle & Gil 2011; Droit-Volet & Gil 2009; Droit-Volet & Meck 2007; Noulhiane et al. 2007; Schirmer 2011; Stetson, Fiesta & Eagleman 2007; van Wassenhove et al. 2011; Yamada & Kawabe 2011). 324 For an analysis of the design of the voices of In the Cut see Macallan & Plain (Macallan & Plain 2010). 325 The opening sequence of Punch-Drunk Love (Anderson, PT 2002) is available on the DVD-ROM which accompanies Appendix D. 326 The ‘Tokyo Tonight’ sequence occurs at approximately 1:02:30–1:12:20 (hh:mm:ss), and is available on the accompanying DVD-ROM to Appendix D. 327 Here I take up Margaret Wilson’s concept of ‘cognitive tools’ (2002, 2008, 2010). 328 Verstehen (interpretive understanding) and Erklärung (causal explanation) demarcate the forms of significance typically privileged within the humanities and sciences respectively. Crudely put, Verstehen values the mental over the physical. Here, then, is the root of the tension between the ‘Two Cultures’ so infamously described by C.P. Snow (Snow 1956). For an explication of the concepts of Verstehen and Erklärung, and the tension between these stances when taken up by the humanities and sciences, see Slingerland (2008a), particularly pages 2–4 and 83. For a consideration of embodiment as a potential approach to conceptually bond mind and body, Verstehen and Erklärung, see Caracciolo (2014). For an extended discussion of consilience and Third Culture see (Brockman 1995; Slingerland 2008a; Wilson, EO 1998). 329 Gallese and Lakoff, for example, contend that “imagining is a form of simulation – a mental simulation of action or perception, using many of the same neurons as actually acting or perceiving” (2005, pp. 456-457, emphasis in original). 330 For a discussion of the precedence of ecological sound in human audition, see Ma & Thompson who claim “speech and music originated from a common emotional signal system based on the imitation and modification of sounds in the environment” (2015, p. 14563). 331 Connected to this assumption which values the mental over the physical are the concepts of Verstehen (interpretive understanding) and Erklärung (causal explanation) which divide the forms of significance typically generated by the humanities and the sciences respectively. 332 For a discussion of the evolutionary tango between biology, culture and technology that “extends our biological capabilities, but is also expected to create selective pressures on these capabilities” see Yadin Dudai (2012). 333 For the role of the brainstem and thalamus in generating affective consciousness see Jaak Panksepp (Panksepp 1998a, 2001, 2005), in particular Solms & Panksepp (Solms & Panksepp 2012). See James Lewis and colleagues (2004) for a discussion of the neural regions recruited for the identification of environmental sounds, biological motion, and action knowledge, with particular regard to the similarities these networks have to counterparts in the visual pathway. 334 For an overview of the relationship between cognitive neuroscience, artistic knowledge and aesthetics see Dustin Stokes (2009), William P. Seeley (2006), Steven Brown and colleagues (2011), Vittorio Gallese and Cinzia Di Dio (2012), Blake Gopnik (2011), and Elisabeth Schellekens (2012). 335 Indeed, there are current signs of a re-awakening of interest in cinematic spectacle. See, for example, Erlend Lavik (2008), Geoff King (King, G 2003), and Aylish Wood (2002). For a fresh take on the spectacle/narrative distinction in Film Studies, although one slightly at odds with the perspective I adopt in this thesis, see Simon Lewis (2014) who posits spectacle and narrative to be interrelated forms of information transmission.

336 Indeed, Alva Noë proposes “art can make a needed contribution to the study of perceptual consciousness” (2000, p. 123), and Yadin Dudai explicitly proposes cinema to be “a scientific experimentation space” (2012). For an overview of thought regarding the relationship between cognitive neuroscience and body–mind dynamics, see Catherine Kerr (2008), and Fiala, Arico & Nichols (2011). 337 For recent examples of this kind of cross-fertilisation of artistic and scientific practice see Heimann (2014). See also Michele Guerra (2015) for a discussion of how cinematic visual movement may be explored for its ecological validity in the activation of the MNS and social cognition, and Tim J. Smith (2013; 2012) for an exploration of audience reception of filmmakers’ intentions through the eye-tracking of actual movies. However, as with much in Film Studies, these examples reveal a focus upon the visual to the exclusion of the auditory, illustrating the need for yet more innovative experiments to capture the role of multimodality in cinematic experience. 338 Including such Deleuzian film scholars as David Rodowick (1997, 2010), Gregory Flaxman (2000), Barbara M. Kennedy (2000), Brian Massumi (2002a, 2002b), Ronald Bogue (2003b, 2003a), Patricia Pisters (2003), Anna Powell (2005, 2007), and Ian Buchanan (2008; 2004). 339 Indeed, Andrew Murphie (2010) describes Deleuze and Guattari’s work as anticipating contemporary findings in cognitive science. 340 Above-the-line personnel refers to individuals who perform such roles as screenwriter, producer and director while below- the-line personnel refers to individuals who create in a direct and hands-on manner the audiovisual imagery such as cinematographer, production designer, editor, sound designer, and composer (to name only a few such roles). By this distinction it is not difficult to grasp that those personnel directly responsible for perceptual design are below-the-line personnel.

------this page intentionally blank ------

24, 2001–2010, television drama, 20th Century Fox Television, USA.

Abbott, HP 2000, ‘The evolutionary origins of the storied mind: modeling the prehistory of narrative consciousness and its discontents’, Narrative, vol. 8, no. 3, pp. 247–56.

---- 2003, ‘Unnarratable knowledge: the difficulty of understanding evolution by natural selection’, in D Herman (ed.), Narrative theory and the cognitive sciences, CSLI Publications, Stanford, Calif., pp. 143–62.

---- 2008, The Cambridge introduction to narrative, 2nd edn, Cambridge University Press, Cambridge.

Abel, R & Altman, R (eds) 2001, The sounds of early cinema, Indiana University Press, Bloomington, IN.

Agnew, ZK, Bhakoo, KK & Puri, BK 2007, ‘The human mirror system: a motor resonance theory of mind- reading’, Brain Research Reviews, vol. 54, no. 2, pp. 286–93.

Ahmed, S 2004, The cultural politics of emotion, Edinburgh University Press, Edinburgh.

---- 2010, ‘Happy objects’, in M Gregg & GJ Seigworth (eds), The affect theory reader, Duke University Press, Durham, NC, pp. 29–51.

Allen, I 1998, ‘Are movies too loud?’, SMPTE Journal, vol. 107, no. 1, pp. 30–44.

Altenmüller, E, Kopiez, R & Grewe, O 2013a, ‘Strong emotions in music: are they an evolutionary adaptation?’, in R Bader (ed.), Sound - Perception - Performance, Springer International Publishing, vol. 1, pp. 131– 56.

---- 2013b, ‘A contribution to the evolutionary basis of music: lessons from the chill response’, in E Altenmüller, S Schmidt & E Zimmermann (eds), Evolution of emotional communication: from sounds in nonhuman mammals to speech and music in man, Oxford University Press, Oxford, pp. 313–35.

Alter, NM 2011, ‘Screening out sound: Arnheim and cinema's silence’, in S Higgins (ed.), Arnheim for film and media studies, Routledge, New York, pp. 69–86.

Altman, R 1975, Nashville, motion picture, USA.

Altman, R 1980a, ‘Introduction: Cinema/Sound’, Yale French Studies, no. 60, pp. 3–15.

---- 1980b, ‘Cinema/Sound’, Yale French Studies, no. 60.

---- 1985, ‘The evolution of sound technology’, in E Weis & J Belton (eds), Film sound: theory and practice, Columbia University Press, New York, pp. 44–53.

---- 1987, The American film musical, Indiana University Press, Bloomington.

---- 1992a, ‘The material heterogeneity of recorded sound’, in R Altman (ed.), Sound theory / sound practice, Routledge, New York, pp. 15–31.

---- 1992b, ‘Afterword: a baker’s dozen of new terms for sound analysis’, in R Altman (ed.), Sound theory / sound practice, Routledge, New York, pp. 249–54.

---- 1992c, ‘General introduction: cinema as event’, in R Altman (ed.), Sound theory / sound practice, Routledge, New York, pp. 1–14.

289 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. ---- 1992d, ‘Introduction: Four and a half film fallacies’, in R Altman (ed.), Sound theory / sound practice, Routledge, New York, pp. 35–45.

Altman, R 1993, Short Cuts, motion picture, USA.

Altman, R 1994, ‘Deep-focus sound: Citizen Kane and the radio aesthetic’, Quarterly Review of Film and Video, vol. 15, no. 3, pp. 1–33.

---- 1996, ‘The silence of the silents’, The Musical Quarterly, vol. 80, no. 4, pp. 648–718.

---- 1999a, ‘Film sound – all of it’, Iris, vol. 27, pp. 31–48.

---- 1999b, Film/Genre, British Film Institute, London.

---- 1999c, ‘Sound Studies: a field whose time has come’, Iris, vol. 27, no. Spring, pp. 3–4.

---- 2004, Silent film sound, Columbia University Press, New York.

---- 2012, ‘Two or three things we thought we knew about silent film sound’, in C Lucia, R Grundmann & A Simon (eds), The Wiley-Blackwell History of American Film, Blackwell Publishing Ltd, Malden, MA, pp. 417–37.

---- 2017, ‘The early cinema soundscape’, in M Mera, R Sadoff & B Winters (eds), The Routledge companion to screen music and sound, Routledge, New York, NY ; London, pp. 190–200.

Altman, R, Jones, M & Tatroe, S 2000, ‘Inventing the cinema soundtrack: Hollywood’s multiplane sound system’, in J Buhler, C Flinn & D Neumeyer (eds), Music and cinema, University Press of New England, Hanover, NH ; London, pp. 339–59.

Altman, R (ed.) 1992e, Sound theory / sound practice, AFI film readers., Routledge, New York.

Ament, VT 2009, The Foley grail: the art of performing sound for film, games, and animation, Focal Press/Elsevier, Amsterdam ; Boston.

Anderson, JD 1993, ‘Sound and image together: cross-modal confirmation’, Wide Angle, vol. 15, no. 1, pp. 30– 43.

---- 1996, The reality of illusion: an ecological approach to a cognitive film theory, Southern Illinois University Press, Carbondale.

Anderson, JD & Fisher, B 1978, ‘The myth of persistence of vision’, Journal of the University Film Association, vol. 30, no. 4 (Topics in film/video/photography theory), pp. 3–8.

Anderson, JD & Anderson, BF 1996, ‘The case for an ecological metatheory’, in D Bordwell & N Carroll (eds), Post-theory: reconstructing film studies, University of Wisconsin Press, Madison, pp. 347–67.

---- (eds) 2005, Moving image theory: ecological considerations, Southern Illinois University Press, Carbondale.

Anderson, ML 2010, ‘Neural reuse: a fundamental organizational principle of the brain’, Behavioral and Brain Sciences vol. 33, no. 4, pp. 245–66.

Anderson, ML & Finlay, BL 2014, ‘Allocating structure to function: the strong links between neuroplasticity and natural selection’, Frontiers in Human Neuroscience, vol. 7, Article 918, pp. 1–16, doi 10.3389/fnhum.2013.00918, viewed 7 January 2014, .

Anderson, PT 1999, Magnolia, motion picture, USA.

---- 2002, Punch-Drunk Love, motion picture, USA.

Anon. 2010, Sound Design Stack Exchange, website, viewed 19 January 2019

---- 2016, ‘Michel Chion’, Article 18 October 2016, viewed 21 November 2018, .

Arieti, S 1976, Creativity: the magic synthesis, Basic Books, New York.

Aristotle c. 335 BCE, Poetics, Penguin classics, Penguin Books, London ; New York, N.Y.

Arnheim, R 1938, ‘A new laocoön: artistic composites and the talking film’, in Film as art, University of California Press, Berkeley, pp. 199–230.

Arnheim, R 1999, ‘Composites of media: the history of an idea’, Michigan Quarterly Review, vol. XXXVIII, no. 4, pp. 558–61.

Arrighi, R, Marini, F & Burr, D 2009, ‘Meaningful auditory information enhances perception of visual biological motion’, Journal of Vision, vol. 9, no. 4, pp. 25.1–7.

Asad, J, Spiller, MJ & Jonas, C 2016, ‘Auditory and visual crossmodal correspondences with haptically perceived liquid viscosity’, Multisensory Research, vol. 29, no. 8, pp. 727–47.

Aschersleben, G & Bertelson, P 2003, ‘Temporal ventriloquism: crossmodal interaction on the time dimension. 2. Evidence from sensorimotor synchronization’, International Journal of Psychophysiology, vol. 50, no. 1-2, pp. 157–63.

Asutay, E, Västfjäll, D, Tajadura-Jiménez, A, Genell, A, Bergman, P & Kleiner, M 2012, ‘Emoacoustics: a study of the psychoacoustical and psychological dimensions of emotional sound design’, Journal of the Audio Engineering Society, vol. 60, no. 1/2, pp. 21–8.

Atwood, M 1996, Alias Grace, Bloomsbury, London.

Averill, JR 1985, ‘The social construction of emotion: with special reference to love’, in KJ Gergen & KE Davis (eds), The social construction of the person, Springer-Verlag, New York, pp. 89–109.

Averill, JR & Nunley, EP 1992, Voyages of the heart: living an emotionally creative life, Free Press, New York.

Baars, BJ 1988, A cognitive theory of consciousness, Cambridge University Press, Cambridge ; New York.

---- 1992, ‘Is consciousness recent?’, Consciousness and Cognition, vol. 1, no. 2, pp. 139–42.

---- 1997, ‘In the theatre of consciousness. Global Workspace Theory, a rigorous scientific theory of consciousness’, Journal of Consciousness Studies, vol. 4, no. 4, pp. 292–309.

---- 2003, ‘The global brainweb: an update on global workspace theory’, Science and Consciousness Review, Article viewed 7 July 2017, .

Bach, DR, Neuhoff, JG, Perrig, W & Seifritz, E 2009, ‘Looming sounds as warning signals: the function of motion cues’, International Journal of Psychophysiology, vol. 74, no. 1, pp. 28–33.

Bach, DR, Schächinger, H, Neuhoff, JG, Esposito, F, Salle, FD, Lehmann, C, Herdener, M, Scheffler, K & Seifritz, E 2008, ‘Rising sound intensity: an intrinsic warning cue activating the amygdala’, Cerebral cortex, vol. 18, no. 1, pp. 145–50.

Bálint, K & Tan, ESH 2015, ‘"It feels like there are hooks inside my chest”: the construction of narrative absorption experiences using image schemata’, Projections, vol. 9, no. 2, pp. 63–88.

291 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Baños, RM, Botella, C, Alcañiz, M, Liaño, V, Guerrero, B & Rey, B 2004, ‘Immersion and emotion: their impact on the sense of presence’, CyberPsychology & Behavior, vol. 7, no. 6, pp. 734–41.

Bar-Haim, Y, Kerem, A, Lamy, D & Zakay, D 2010, ‘When time slows down: the influence of threat on time perception in anxiety’, Cognition and Emotion, vol. 24, no. 2, pp. 255–63.

Baranowski, AM & Hecht, H 2016, ‘The auditory Kuleshov effect: multisensory integration in movie editing’, Perception, vol. 46, no. 5, pp. 624–31.

Bargh, JA 1994, ‘The four horsemen of automaticity: awareness, efficiency, intention, and control in social cognition’, in RS Wyer & TK Srull (eds), Handbook of social cognition, 2nd edn, Lawrence Erlbaum Associates, Hillsdale, N.J., pp. 1–40.

---- 1997, ‘The automaticity of everyday life’, in RS Wyer, Jr. (ed.), Advances in social cognition, Lawrence Erlbaum Associates, Mahwah, NJ, vol. X, pp. 1–61.

---- 2011, ‘Unconscious thought theory and its discontents: a critique of the critiques’, Social Cognition, vol. 29, no. 6, pp. 629–47.

Bargh, JA & Chartrand, TL 1999, ‘The unbearable automaticity of being’, American Psychologist, vol. 54, no. 7, pp. 462–79.

Bargh, JA & Morsella, E 2008, ‘The unconscious mind’, Perspectives on Psychological Science, vol. 3, no. 1, pp. 73–9.

Barker, JM 2009, The tactile eye: touch and the cinematic experience, University of California Press, Berkeley.

---- 2011, ‘Touch and the cinematic experience’, in F Bacci & D Melcher (eds), Art and the senses, Oxford University Press, Oxford ; New York, pp. 149–59.

---- 2016, ‘Haunted phenomenology and synesthetic cinema’, Studia Phaenomenologica, vol. 16, no. Film and Phenomenology, pp. 373–408.

Barrett, LF & Wager, TD 2006, ‘The structure of emotion. Evidence from neuroimaging studies’, Current Directions in Psychological Science, vol. 15, no. 2, pp. 79–83.

Barrett, LF, Ochsner, KN & Gross, JJ 2007, ‘On the automaticity of emotion’, in JA Bargh (ed.), Social psychology and the unconscious: the automaticity of higher mental processes, Psychology Press, New York, pp. 173–217.

Barsalou, LW 1999, ‘Perceptual symbol systems’, Behavioral and Brain Sciences, vol. 22, no. 4, pp. 577–660.

---- 2003a, ‘Situated simulation in the human conceptual system’, Language and Cognitive Processes, vol. 18, no. 516, pp. 513–62.

---- 2003b, ‘Abstraction in perceptual symbol systems’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 358, no. 1435, pp. 1177–87.

---- 2005, ‘Continuity of the conceptual system across species’, Trends in Cognitive Sciences, vol. 9, no. 7, pp. 309–11.

---- 2008, ‘Grounded cognition’, Annual Review of Psychology, vol. 59, no. 1, pp. 617–45.

---- 2009, ‘Simulation, situated conceptualization, and prediction’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, pp. 1281–9.

Barsalou, LW, Solomon, KO & Wu, L-L 1999, ‘Perceptual simulation in conceptual tasks’, Amsterdam Studies in the Theory and History of Linguistic Science, no. 152, pp. 209–28.

292 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Barsalou, LW, Simmons, WK, Barbey, AK & Wilson, CD 2003, ‘Grounding conceptual knowledge in modality- specific systems’, Trends in Cognitive Sciences, vol. 7, no. 2, pp. 84–91.

Barthes, R 1968, ‘The reality effect’, in The rustle of language, Hill and Wang, New York, pp. 141–8.

---- 1977a, ‘Introduction to the structural analysis of narratives’, in Image, music, text, Fontana Press, London, pp. 79–124.

---- 1977b, Image, music, text, Fontana Press, London.

---- 1977c, ‘Rhetoric of the image’, in Image, music, text, Fontana Press, London, pp. 32–51.

Bartsch, A 2004, Emotional communication - a theoretical model, paper presented at IGEL 2004: 9th International Congress, Edmonton, Canada, 3–7 August 2004.

---- 2005, A three level approach to meta-emotion, paper presented at Audiovisual Emotions: Representing and Eliciting Emotions through Audiovisual Media Conference, Hamburg, Germany, 1–3 December 2005.

---- 2008, ‘Meta-emotion: how films and music videos communicate emotions about emotions’, Projections, vol. 2, no. 1, pp. 45–59.

---- 2010, ‘Vivid abstractions: on the role of emotion metaphors in film viewers' search for deeper insight and meaning’, Midwest Studies In Philosophy, vol. 34, no. 1, pp. 240–60.

---- 2012, ‘Emotional gratification in entertainment experience. Why viewers of movies and television series find it rewarding to experience emotions’, Media Psychology, vol. 15, no. 3, pp. 267–302.

---- 2016, ‘Embodied emotion metaphors in moving images’, in K Fahlenbrach (ed.), Embodied metaphors in film, television, and video games: cognitive approaches, Routledge, New York ; London, pp. 202–17.

Bartsch, A & Hübner, S 2005, ‘Towards a theory of emotional communication’, Comparative Literature and Culture, vol. 7, no. 4, Article 2, pp. 1–8, doi 10.7771/1481-4374.1278.

Bartsch, A & Oliver, MB 2011, ‘Making sense of entertainment: on the interplay of emotion and cognition in entertainment experience’, Journal of Media Psychology: Theories, Methods, and Applications, vol. 23, no. 1, pp. 12–7.

Bartsch, A & Hartmann, T 2017, ‘The role of cognitive and affective challenge in entertainment experience’, Communication Research, vol. 44, no. 1, pp. 29–53.

Bartsch, A, Kalch, A & Oliver, MB 2014, ‘Moved to think. The role of emotional media experiences in stimulating reflective thoughts’, Journal of Media Psychology, vol. 26, no. 3, pp. 125–40.

Bastiaansen, JACJ, Thioux, M & Keysers, C 2009, ‘Evidence for mirror systems in emotions’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, no. 1528, pp. 2391–404.

Bau, O 2012, REVEL: programming the sense of touch, Disney Research, viewed 7 August 2012, .

Bau, O & Poupyrev, I 2012, ‘REVEL: tactile feedback technology for augmented reality’, ACM Transactions on Graphics, vol. 31, no. 4 pp. 89–100.

Bau, O, Poupyrev, I, Israr, A & Harrison, C 2010, TeslaTouch: electrovibration for touch surfaces, paper presented at UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology, New York, NY, 3–6 October 2010.

Baudry, J-L 1986, ‘Ideological effects of the basic cinematographic apparatus [1970]’, in P Rosen (ed.), Narrative, apparatus, ideology: a film theory reader, Columbia University Press, New York, pp. 286– 98.

293 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Baumeister, RF 2005, The cultural animal: human nature, meaning, and social life, Oxford University Press, New York, NY.

---- 2011, ‘The unity of self at the interface of the animal body and the cultural system’, Psychological Studies, vol. 56, no. 1, pp. 5–11.

Baumeister, RF & Masicampo, EJ 2010, ‘Conscious thought is for facilitating social and cultural interactions: how mental simulations serve the animal-culture interface’, Psychological Review, vol. 117, no. 3, pp. 945–71.

Baumeister, RF, Masicampo, EJ & Vohs, KD 2011, ‘Do conscious thoughts cause behavior?’, Annual Review of Psychology, vol. 62, pp. 331–61.

Baumgartner, T, Esslen, M & Jäncke, L 2006, ‘From emotion perception to emotion experience: emotions evoked by pictures and classical music’, International Journal of Psychophysiology, vol. 60, no. 1, pp. 34–43.

Baumgartner, T, Lutz, K, Schmidt, CF & Jäncke, L 2006, ‘The emotional power of music: how music enhances the feeling of affective pictures’, Brain Research, vol. 1075, no. 1, pp. 151–64.

Bazin, A 1967, What is cinema?, vol. 1, 2 vols., University of California Press, Berkeley.

Bechara, A & Damasio, AR 2005, ‘The somatic marker hypothesis: a neural theory of economic decision’, Games and Economic Behavior, vol. 52, no. 2, pp. 336–72.

Bechara, A, Damásio, AR, Damásio, H & Anderson, SW 1994, ‘Insensitivity to future consequences following damage to human prefrontal cortex’, Cognition, vol. 50, no. 1-3, pp. 7–15.

Bechara, A, Damasio, H, Tranel, D & Damasio, AR 2005, ‘The Iowa Gambling Task and the somatic marker hypothesis: some questions and answers’, Trends in Cognitive Sciences, vol. 9, no. 4, pp. 159–62.

Beck, J 2008, ‘The sounds of “silence”: Dolby Stereo, sound design, and The Silence of the Lambs’, in J Beck & T Grajeda (eds), Lowering the boom: critical studies in film sound, University of Illinois Press, Urbana, pp. 68–83.

---- 2016, Designing sound: audiovisual aesthetics in 1970s American cinema, Rutgers University Press, New Brunswick, New Jersey.

Beck, J & Grajeda, T 2008a, ‘The future of film sound studies’, in J Beck & T Grajeda (eds), Lowering the boom: critical studies in film sound, University of Illinois Press, Urbana, pp. 1–20.

---- (eds) 2008b, Lowering the boom: critical studies in film sound, University of Illinois Press, Urbana.

Belton, J 1999, ‘Awkward transitions: Hitchcock's "Blackmail" and the dynamics of early film sound’, The Musical Quarterly, vol. 83, no. 2, pp. 227–46.

Bergman, P, Sköld, A, Västfjäll, D & Fransson, N 2009, ‘Perceptual and emotional categorization of sound’, Journal of Acoustical Society of America, vol. 126, no. 6, pp. 3156–67.

Bergman, P, Västfjäll, D, Asutay, E, Tajadura, A, Sköld, A, Genell, A & Fransson, N 2008, Emoacoustics: sound character versus source meaning in emotional responses to sounds, paper presented at Performance: 9th International Congress on Noise as a Public Health Problem (ICBEN), Foxwoods, CT.

Bernhardt, C 1947, Possessed, motion picture, USA.

Berridge, K & Kringelbach, M 2008, ‘Affective neuroscience of pleasure: reward in humans and animals’, Psychopharmacology, vol. 199, no. 3, pp. 457–80.

294 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Berridge, KC 2010, ‘Remembering Robert Zajonc: the complete psychologist’, Emotion Review, vol. 2, no. 4, pp. 348–52.

Berridge, KC & Winkielman, P 2003, ‘What is an unconscious emotion? (The case for unconscious "liking")’, Cognition and Emotion, vol. 17, no. 2, pp. 181–211.

Bertelson, P 1999, ‘Ventriloquism: a case of crossmodal perceptual grouping’, in G Aschersleben, T Bachmann & J Müsseler (eds), Cognitive contributions to the perception of spatial and temporal events, Elsevier, Amsterdam, vol. 129, pp. 347–62.

Bertelson, P & Aschersleben, G 1998, ‘Automatic visual bias of perceived auditory location’, Psychonomic Bulletin & Review, vol. 5, no. 3, pp. 482–9.

---- 2003, ‘Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory– visual temporal order judgment’, International Journal of Phsychophysiology, vol. 50, no. 1-2, pp. 147– 55.

Biancorosso, G 2016, Situated listening: the sound of absorption in classical cinema, Oxford University Press, New York, NY.

Bickart, KC, Wright, CI, Dautoff, RJ, Dickerson, BC & Barrett, LF 2011, ‘Amygdala volume and social network size in humans’, Nature Neuroscience, vol. 14, no. 2, pp. 163–4.

Bijsterveld, K (ed.) 2013, Soundscapes of the urban past: staged sound as mediated cultural heritage, Transcript-Verlag, Bielefeld, Germany.

Biocca, F & Delaney, B 1995, ‘Immersive virtual reality technology’, in F Biocca & MR Levy (eds), Communication in the age of virtual reality, Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 57–124.

Birdsall, C & Enns, A (eds) 2008, Sonic mediations: body, sound, technology, Cambridge Scholars Publishing, Newcastle.

Bizzaro, N 2011, ‘Pierre Schaeffer’s contribution to audiovisual theory’, Worlds of Audio Vision, Article 2, viewed 5 January 2014, .

Blair, R 2006, ‘Image and action: cognitive neuroscience and actor training’, in BA McConachie & FE Hart (eds), Performance and cognition: theatre studies and the cognitive turn, Routledge, New York; London, pp. 167–85.

---- 2007, ‘(Refuting) Arguments for the end of theatre: possible implications of cognitive neuroscience for performance’, Journal of Dramatic Theory and Criticism, vol. XXI, no. 2, pp. 125–34.

---- 2008, The actor, image, and action: acting and cognitive neuroscience, Routledge, New York; London.

---- 2009a, ‘Cognitive neuroscience and acting: imagination, conceptual blending, and empathy’, The Drama Review vol. 53, no. 4, pp. 92–103.

---- 2009b, ‘Cognitive neuroscience and acting: imagination, conceptual blending, and empathy’, The Drama Review, vol. 53, no. 4, pp. 93–103.

Block, N 1995, ‘On a confusion about a function of consciousness’, Behavioral and Brain Sciences, vol. 18, no. 2, pp. 227–87.

---- 2007a, ‘Overflow, access, and attention’, Behavioral and Brain Sciences, vol. 30, no. 5/6, pp. 530–48.

---- 2007b, ‘Consciousness, accessibility, and the mesh between psychology and neuroscience’, Behavioral and Brain Sciences, vol. 30, no. 5/6, pp. 481–548.

---- 2009, ‘Comparing the major theories of consciousness’, in MS Gazzaniga (ed.), The cognitive neurosciences, 4th edn, MIT Press, Cambridge, Mass., pp. 1111–22.

---- 2011a, ‘The Higher Order approach to consciousness is defunct’, Analysis, vol. 71, no. 3, pp. 419–31.

---- 2011b, ‘Perceptual consciousness overflows cognitive access’, Trends in Cognitive Sciences, vol. 15, no. 12, pp. 567–75.

Blood, AJ & Zatorre, RJ 2001, ‘Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion’, Proceedings of the National Academy of Sciences of the United States of America, vol. 98, no. 20, pp. 11818–23.

Blood, AJ, Zatorre, RJ, Bermudez, P & Evans, AC 1999, ‘Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions’, Nature Neuroscience, vol. 2, no. 4, pp. 382–7.

Bogue, R 2003a, Deleuze on music, painting, and the arts, Routledge, New York ; London.

---- 2003b, Deleuze on cinema, Deleuze and the arts, Routledge, New York London.

Boone, AR 1933, ‘Prehistoric monsters roar and hiss for sound film’, Popular Science, no. 122, pp. 20–1 and 106.

Bordwell, D 1985, Narration in the fiction film, University of Wisconsin Press, Madison, Wisconsin.

---- 1989a, Making meaning: inference and rhetoric in the interpretation of cinema, Cambridge, Mass., Harvard University Press.

---- 1989b, ‘Historical poetics of cinema’, in RB Palmer (ed.), The cinematic text: methods and approaches, AMS Press, New York, vol. 3, pp. 369–98.

---- 1989c, ‘A case for cognitivism’, Iris, vol. 9, no. Spring, pp. 11–40.

---- 1990, ‘A case for cognitivism: further reflections’, Iris, vol. 11, no. Summer, pp. 107–12.

---- 1996, ‘Contemporary film studies and the vicissitudes of Grand Theory’, in D Bordwell & N Carroll (eds), Post-theory: reconstructing film studies, University of Wisconsin Press, Madison, pp. 3–36.

---- 2002, ‘Intensified continuity visual style in contemporary American film’, Film Quarterly, vol. 55, no. 3, pp. 16–28.

---- 2006, The way Hollywood tells it: story and style in modern movies, University of California Press, Berkeley.

---- 2008, Poetics of cinema, Routledge, New York.

Bordwell, D, Staiger, J & Thompson, K 1985, The classical Hollywood cinema: film style & mode of production to 1960, Routledge, London.

Bordwell, D & Carroll, N (eds) 1996, Post-theory: reconstructing film studies, Wisconsin studies in film., University of Wisconsin Press, Madison.

Bottomore, S 1999, ‘An international survey of sound effects in early cinema’, Film History, vol. 11, no. 4, pp. 485–98.

---- 2001, ‘The story of Percy Peashaker: debates about sound effects in the early cinema’, in R Abel & R Altman (eds), The sounds of early cinema, Indiana University Press, Bloomington, IN, pp. 129–42.

Bowser, E 1994, The transformation of cinema, 1907-1915, History of the American cinema, University of California Press, Berkeley.

Boyd, R & Richerson, PJ 1985, Culture and the evolutionary process, University of Chicago Press, Chicago.

296 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. ---- 2005, The origin and evolution of cultures, Evolution and cognition, Oxford University Press, New York.

Boyle, D 2007, Sunshine, motion picture, UK, USA.

Brain, RM 2012, ‘Self-projection: Hugo Münsterberg on empathy and oscillation in cinema spectatorship’, Science in Context, vol. 25, no. 3, pp. 329–53.

Branigan, E 1992a, Narrative comprehension and film, Sightlines, Routledge, London ; New York.

---- 1992b, ‘Narrative schema’, in Narrative comprehension and film, Routledge, London ; New York, pp. 1–32.

Bråten, S (ed.) 2007, On being moved: from mirror neurons to empathy, John Benjamins Publishing Company, Amsterdam.

Brattico, E & Pearce, M 2013, ‘The neuroaesthetics of music’, Psychology of Aesthetics, Creativity, and the Arts, vol. 7, no. 1, pp. 48–61.

Bregman, AS 1990, Auditory scene analysis: the perceptual organization of sound, A Bradford book, MIT Press, Cambridge, Mass.

---- 2002, ‘The auditory scene’, in DJ Levitin (ed.), Foundations of cognitive psychology: core readings, MIT Press, Cambridge, MA, pp. 213–48.

---- 2008, ‘Auditory scene analysis’, in AI Basbaum, A Kaneko, GM Shepherd & G Westheimer (eds), The senses: a comprehensive reference, Academic Press, San Diego, vol. 3, pp. 861–70.

Bremner, AJ, Holmes, NP & Spence, C 2012, ‘The development of multisensory representations of the body and of the space around the body’, in AJ Bremner, DJ Lewkowicz & C Spence (eds), Multisensory development, Oxford University Press, Oxford, pp. 113–36.

Bremner, AJ, Lewkowicz, DJ & Spence, C (eds) 2012, Multisensory development, Oxford University Press, Oxford.

Bresson, R 1986, Notes on the cinematographer, Quartet Books, London.

Brewer, WF 1995, ‘To assert that essentially all human knowledge and memory is represented in terms of stories is certainly wrong’, in RS Wyer Jr. (ed.), Knowledge and memory: the real story, Lawrence Erlbaum Associates, Hillsdale, N.J., pp. 109–20.

Brewer, WF & Lichtenstein, EH 1981, ‘Event schemas, story schemas, and story grammars’, in J Long & AD Baddeley (eds), Attention and performance, Erlbaum, Hillsdale, NJ, vol. 9, pp. 363–79.

Brockman, J (ed.) 1995, The third culture: beyond the scientific revolution, Simon & Schuster, New York.

Brophy, P 2004, 100 modern soundtracks, BFI, London.

---- 2008, ‘Where sound is: locating the absent aural in film theory’, in J Donald & M Renov (eds), The SAGE Handbook of Film Studies, SAGE, Los Angeles ; London, pp. 424–35.

---- (ed.) 1999, Cinesonic: the world of sound in film, Cinesonic, Australian Film, Television and Radio School, Sydney.

---- (ed.) 2000, Cinesonic 2: cinema and the sound of music, Australian Film Television and Radio School, Sydney.

---- (ed.) 2001, Cinesonic 3: experiencing the soundtrack, Cinesonic, Australian Film, Television and Radio School, Sydney.

Brown, R 2012, ‘The myth of phenomenological overflow’, Consciousness and Cognition, vol. 21, no. 2, pp. 599–604.

Brown, RS 1994, Overtones and undertones: reading film music, University of California Press, Berkeley.

Brown, S, Gao, X, Tisdelle, L, Eickhoff, SB & Liotti, M 2011, ‘Naturalizing aesthetics: brain areas for aesthetic appraisal across sensory modalities’, NeuroImage, vol. 58, no. 1, pp. 250–8.

Bruno, G 2009, ‘Film, aesthetics, science: Hugo Münsterberg's laboratory of moving images’, Grey Room, no. 36, pp. 88–113.

Bryant, J & Miron, D 2003, ‘Excitation-transfer theory and three-factor theory of emotion’, in D Roskos- Ewoldsen, J Bryant & J Cantor (eds), Communication and emotion: Essays in honor of Dolf Zillmann, Lawrence Erlbaum Associates, Mahwah, NJ, pp. 31–59.

Buchanan, I & MacCormack, P 2008, Deleuze and the schizoanalysis of cinema, Continuum, London ; New York.

Buchanan, I & Swiboda, M (eds) 2004, Deleuze and music, Edinburgh University Press, Edinburgh.

Bulgakowa, O 2006, ‘The ear against the eye: Vertov's Symphony’, Monatshefte, vol. 98, no. 2, pp. 219–39.

Bull, M, Gilroy, P, Howes, D & Kahn, D 2006, ‘Introducing Sensory Studies’, The Senses and Society, vol. 1, no. 1, pp. 5–7.

Bull, M & Back, L (eds) 2003, The auditory culture reader, Sensory formations series., Berg, Oxford.

Bushman, BJ 2002, ‘Does venting anger feed or extinguish the flame? Catharsis, rumination, distraction, anger, and aggressive responding’, Personality and Social Psychology Bulletin, vol. 28, no. 6, pp. 724–31.

Cacioppo, JT 1998, ‘Somatic responses to psychological stress: the reactivity hypothesis’, in M Sabourin (ed.), Advances in Psychological Science, Psychology Press, Hove, vol. 2, pp. 87–114.

Calvert, GA, Bullmore, ET, Brammer, MJ, Campbell, R, Williams, SCR, McGuire, PK, Woodruff, PWR, Iversen, SD & David, AS 1997, ‘Activation of auditory cortex during silent lipreading’, Science, vol. 276, no. 5312, pp. 593–6.

Calvert, GA, Spence, C & Stein, BE (eds) 2004, The handbook of multisensory processes, MIT Press, Cambridge, Mass.

Calvo-Merino, B, Jola, C, Glaser, DE & Haggard, P 2008, ‘Towards a sensorimotor aesthetics of performing art’, Consciousness and Cognition, vol. 17, no. 3, pp. 911–22.

Cameron, J 1997, Titanic, motion picture, USA.

Campbell, V 2012, "Mind in Life" with Evan Thompson (BSP 89), viewed 11 September 2013

Campion, J 1990, An Angel at My Table, motion picture, New Zealand, Australia, UK.

---- 1993, The Piano, motion picture, Australia, New Zealand, France.

---- 1996, The Portrait of a Lady, motion picture, UK, USA.

---- 2003, In the Cut, motion picture, Australia, USA, UK.

---- 2009, Bright Star, motion picture, UK, Australia, France.

Candusso, D 2016a, The immersive cinematic sound space: audience perspectives, paper presented at ASPERA Annual Conference 2016: Screen Production Research: The Big Questions, Canberra, Australia, 5–7 July 2016.

298 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. ---- 2016b, ‘Digital 3D and the contemporary soundtrack: competing cinema spaces’, The New Soundtrack, vol. 6, no. 1, pp. 63–76.

Cánovas, CP 2010, Conceptual blending theory and the history of emotions, paper presented at CogSci 2010, Portland, Oregon, 11–14 August 2010.

Caracciolo, M 2012, ‘Narrative, meaning, interpretation: an enactivist approach’, Phenomenology and the Cognitive Sciences, vol. 11, no. 3, pp. 1–18.

---- 2014, ‘Interpretation for the bodies: bridging the gap’, Style, vol. 48, no. 3, pp. 385–403.

Caramiaux, B, Françoise, J, Schnell, N & Bevilacqua, F 2014, ‘Mapping through listening’, Computer Music Journal, vol. 38, no. 3, pp. 34–48.

Carlsson, SE 1996, Filmsound, Sweden, viewed 19 January 2019, .

Carroll, B 2016, Feeling film: a spatial approach, Palgrave Macmillan, Basingstoke, UK.

Carroll, N 1988, ‘Film/mind analogies: the case of Hugo Münsterberg’, Journal of Aesthetics and Art Criticism, vol. 46, no. 4, pp. 489–99.

---- 1996, ‘Prospects for film theory: a personal assessment’, in D Bordwell & N Carroll (eds), Post-theory: reconstructing film studies, University of Wisconsin Press, Madison, pp. 37–70.

Carroll, Nl 1982, ‘Address to the Heathen’, October, vol. 23, no. Winter, pp. 89–163.

---- 1985, ‘The specificity of media in the arts’, Journal of Aesthetic Education, vol. 19, no. 4, pp. 5–20.

---- 1988a, Philosophical problems of classical film theory, Princeton University Press, Princeton, N.J.

---- 1988b, Mystifying movies: fads & fallacies in contemporary film theory, Columbia University Press, New York.

---- 1990, The philosophy of horror, or paradoxes of the heart, Routledge, New York.

---- 1996, ‘Cracks in the acoustic mirror’, in Theorizing the moving image, Cambridge University Press, New York, pp. 336–42.

---- 1997, ‘Art, narrative, and emotion’, in M Hjort & S Laver (eds), Emotion and the arts, Oxford University Press, New York, pp. 190–211.

---- 1999, ‘Film, emotion, and genre’, in CR Plantinga & GM Smith (eds), Passionate views: film, cognition, and emotion, Johns Hopkins University Press, Baltimore, pp. 21–48.

Carvalho, FR, Wang, Q, Van Ee, R & Spence, C 2016, ‘The influence of soundscapes on the perception and evaluation of beers’, Food Quality and Preference, vol. 52, pp. 32–41.

Cavalli-Sforza, LL & Feldman, MW 1981, Cultural transmission and evolution: a quantitative approach, Monographs in population biology, Princeton University Press, Princeton, N.J.

Chalmers, DJ 1995, ‘Facing up to the problem of consciousness’, Journal of Consciousness Studies, vol. 2, no. 3, pp. 200–19.

Chanan, M 1980, The dream that kicks: the prehisotry and early years of cinema in Britain, Routledge & Kegan Paul, London.

Charland, LC 1999, ‘Perceptual symbol systems and emotion’, Behavioral and Brain Sciences, vol. 22, pp. 612– 3.

299 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Chartrand, TL, van Baaren, RB & Bargh, JA 2006, ‘Linking automatic evaluation to mood and information processing style: consequences for experienced affect, impression formation, and stereotyping’, Journal of Experimental Psychology, vol. 135, no. 1, pp. 70–7.

Chiao, JY 2009a, ‘Cultural neuroscience: a once and future discipline’, Progress in Brain Research, vol. 178, pp. 287–304.

Chiao, JY & Ambady, N 2007, ‘Cultural neuroscience: parsing universality and diversity across levels of analysis’, in S Kitayama & D Cohen (eds), The handbook of cultural psychology, Guilford Press, New York, pp. 237–54.

Chiao, JY (ed.) 2009b, Cultural neuroscience: cultural influences on brain function, Progress in Brain Research, Elsevier, New York; Oxford; Amsterdam.

Chion, M 1982, La voix au cinéma, Editions de l'Etoile, Paris.

---- 1985, Le son au cinéma, Editions de l'Etoile, Paris.

---- 1988, La Toile trouée, Cahiers du cinéma, Paris.

---- 1994, Audio-vision: sound on screen, Columbia University Press, New York.

---- 1999, The voice in cinema, Columbia University Press, New York.

---- 2003, ‘The silence of the loudspeakers, or why with Dolby sound it is the film that listens to us’, in L Sider, D Freeman & J Sider (eds), Soundscape: the School of Sound lectures, 1998-2001, Wallflower Press, London, pp. 150–4.

---- 2009, Film, a sound art, English edn, Columbia University Press, New York ; Chichester.

---- 2012, 100 concepts to think and describe sound cinema, viewed 15 January 2015, .

---- 2013, ‘Sensory aspects of contemporary cinema’, in J Richardson, C Gorbman & C Vernallis (eds), The Oxford Handbook of New Audiovisual Aesthetics, Oxford University Press, Oxford ; New York, pp. 325–30.

---- 2016, Sound: an acoulogical treatise, Duke University Press, Durham.

Chion, M & Brewster, B 1991, ‘Quiet revolution... and rigid stagnation’, October, vol. 58, pp. 69–80.

Churchland, PS, Ramachandran, VS & Sejnowski, TJ 1994, ‘A critique of pure vision’, in C Koch & JL Davis (eds), Large scale neuronal theories of the brain, MIT Press, Cambridge, Mass., pp. 23–60.

Clair, R 1985 [1929], ‘The art of sound’, in E Weis & J Belton (eds), Film sound: theory and practice, Columbia University Press, New York, pp. 92–5.

Clark, A 1997, Being there: putting brain, body, and world together again, MIT Press, Cambridge, Mass.

---- 1999, ‘An embodied cognitive science?’, Trends in Cognitive Sciences, vol. 3, no. 9, pp. 345–51.

---- 2007, ‘A sense of presence’, Pragmatics and Cognition, vol. 15, no. 3, pp. 413–33.

Cleeremans, A & Sarrazin, J-C 2007, ‘Time, action, and consciousness’, Human Movement Science, vol. 26, no. 2, pp. 180–202.

Clore, GL & Ortony, A 2000, ‘Cognition in emotion: always, sometimes, or never?’, in L Nadel & R Lane (eds), The cognitive neuroscience of emotion, Oxford University Press, Oxford ; New York, pp. 24–61.

300 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Clore, GL & Ortony, A 2008, ‘Appraisal theories: how cognition shapes affect into emotion’, in M Lewis, JM Haviland-Jones & LF Barrett (eds), Handbook of emotions, 3rd edn, Guilford Press, New York, pp. 626–42.

Clore, GL, Storbeck, J, Robinson, MD & Centerbar, DB 2005, ‘Seven sins in the study of unconscious affect’, in LF Barrett, P Niedenthal & P Winkielman (eds), Emotion and consciousness, The Guilford Press, New York, pp. 384–408.

Clough, PT 2007, ‘Introduction’, in PT Clough & J Halley (eds), The affective turn: theorizing the social, Duke University Press, Durham, N.C., pp. 1–33.

Cobussen, M, Meelberg, V & Truax, B (eds) 2017, The Routledge companion to sounding art, Routledge, New York ; London.

Cohen, AJ 1990, ‘Understanding musical soundtracks’, Empirical Studies of the Arts, vol. 8, no. 2, pp. 111–24.

---- 1998, The functions of music in multimedia: a cognitive approach, paper presented at Fifth International Conference on Music Perception and Cognition, Seoul, Korea, 26–30 August 1998.

---- 2001, ‘Music as a source of emotion in ﬁlm’, in PN Juslin & JA Sloboda (eds), Music and emotion: theory and research, Oxford University Press, pp. 249–72.

---- 2005, ‘How music influences the interpretation of film and video: approaches from experimental psychology’, in RA Kendall & RWH Savage (eds), Selected reports in ethnomusicology: perspectives in systematic musicology, University of California Press, Los Angeles, CA, vol. 12, pp. 15–36.

---- 2013a, ‘Film music and the unfolding narrative’, in MA Arbib (ed.), Language, music, and the brain: a mysterious relationship, MIT Press, Cambridge, MA, pp. 173–201.

---- 2013b, ‘Congruence-association model of music and multimedia: origin and evolution’, in S-L Tan, A Cohen, SD Lipscomb & RA Kendall (eds), The psychology of music in multimedia, Oxford University Press, Oxford, UK, pp. 17–47.

---- 2014a, ‘Resolving the paradox of film music through a cognitive narrative approach to film comprehension’, in JC Kaufman & DK Simonton (eds), The social science of cinema, Oxford University Press, New York, NY, pp. 57–83.

---- 2014b, ‘Film music from the perspective of cognitive science’, in D Neumeyer (ed.), The Oxford Handbook of Film Music Studies, Oxford University Press, Oxford ; New York, pp. 96–130.

---- 2015, ‘Congruence-association model and experiments in film music: toward interdisciplinary collaboration’, Music and the Moving Image, vol. 8, no. 2, pp. 5–24.

Colavita, FB 1974, ‘Human sensory dominance (auditory versus visual stimulus)’, Perception & Psychophysics, vol. 16, no. 2, pp. 409–12.

Coleman, M 2009, Soundworks Collection, website, viewed 18 January 2019

---- 2011, The sound of Inception, online multimedia, viewed 28 January 2013

---- 2015, Coll Anderson on documentary sound design, online multimedia, viewed 21 October 2016

Collier, GL 1996, ‘Affective synesthesia: extracting emotion space from simple perceptual stimuli’, Motivation and Emotion, vol. 20, no. 1, pp. 1–32.

Collignon, O, Girard, S, Gosselin, F, Roy, S, Saint-Amour, D, Lassonde, M & Lepore, F 2008, ‘Audio-visual integration of emotion expression’, Brain Research, vol. 1242, pp. 126–35.

Collins, B 2014, ‘3 million Aussies visited the top two torrent sites in May’, Gizmodo Australia, 23 June 2014.

Colombetti, G 2010, ‘Enaction, sense-making and emotion’, in J Stewart, O Gapenne & EA Di Paolo (eds), Enaction: towards a new paradigm for cognitive science, MIT Press, Cambridge, MA, pp. 145–64.

---- 2013, ‘Some ideas for the integration of neurophenomenology and affective neuroscience’, Constructivist Foundations, vol. 8, no. 3, pp. 288–97.

---- 2014, The feeling body: affective science meets the enactive mind, MIT Press, Cambridge, MA.

Colombetti, G & Thompson, E 2008, ‘The feeling body: toward an enactive approach to emotion’, in WF Overton, U Mueller & JL Newman (eds), Body in mind, mind in body: developmental perspectives on embodiment and consciousness, Lawrence Erlbaum Associates, New York, pp. 45–68.

Connolly, WE 2002, Neuropolitics: thinking, culture, speed, University of Minnesota Press, Minneapolis, MN.

Cook, A 2006, ‘Staging nothing: Hamlet and cognitive science’, SubStance, vol. 35, no. 110, pp. 83–99.

---- 2007, ‘Interplay: the method and potential of a cognitive scientific approach to theatre’, Theatre Journal, vol. 59, no. 4, pp. 579–94.

---- 2010, Shakespearean neuroplay: reinvigorating the study of dramatic texts and performance through cognitive science, Palgrave MacMillan, New York.

Cooper, MC & Schoedsack, EB 1933, King Kong, motion picture, USA.

Coppola, FF 1969, The Rain People, motion picture, USA.

---- 1974a, The Conversation, motion picture, USA.

---- 1974b, The Godfather: Part II, motion picture, USA.

---- 1979, Apocalypse Now, motion picture, USA.

Corbella, M 2010, ‘Sound design. Ambiguity and historical necessity of a fashionable label’, Worlds of Audio Vision, Article 3, viewed 17 February 2011, .

Corne, J 2016, ‘Michel Chion: listening to cinema’, in M Pomerance & RB Palmer (eds), Thinking in the dark: cinema, theory, practice, Rutgers University Press, New Jersey, pp. 229–40.

Coulthard, L 2012, ‘Haptic aurality: listening to the films of Michael Haneke’, Film-Philosophy, vol. 16, no. 1, pp. 16–29.

---- 2013, ‘Dirty sound. Haptic noise in new extremism’, in C Vernallis, A Herzog & J Richardson (eds), The Oxford Handbook of Sound and Image in Digital Media, Oxford University Press, Oxford ; New York, pp. 115–26.

Coutinho, E, Miranda, ER & Cangelosi, A 2005, ‘Towards a model for embodied emotions’, paper presented to 2005 Purtuguese Conference on Artificial Intelligence, Covilhã, Portugal, 5–8 December 2005.

Coutrot, A, Guyader, N, Ionescu, G & Caplier, A 2012, ‘Influence of soundtrack on eye movements during video exploration’, Journal of Eye Movement Research, vol. 5, no. 4, pp. 1–10.

Cowan, N 2010, ‘The magical mystery four: how is working memory capacity limited, and why?’, Current Directions in Psychological Science, vol. 19, no. 1, pp. 51–7.

Cowley, SJ 2004, ‘Simulating others: the basis of human cognition?’, Language Sciences, vol. 26, no. 3, pp. 273–99.

Cox, C & Warner, D (eds) 2004, Audio culture: readings in modern music, Continuum, New York.

Crafton, D 1995, ‘Pie and chase: gag, spectacle and narrative in slapstick comedy’, in KB Karnick & H Jenkins (eds), Classical Hollywood comedy, Routledge, New York, pp. 106–19.

---- 1997, The talkies: American cinema's transition to sound, 1926-1931, Scribner, New York.

Craig, AD 2002, ‘How do you feel? Interoception: the sense of the physiological condition of the body’, Nature Reviews Neuroscience, vol. 3, no. 8, pp. 655–66.

---- 2003, ‘Interoception: the sense of the physiological condition of the body’, Current Opinion in Neurobiology, vol. 13, no. 4, pp. 500–5.

---- 2009, ‘Emotional moments across time: a possible neural basis for time perception in the anterior insula’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, no. 1525, pp. 1933– 42.

Crick, F 1994, The astonishing hypothesis: the scientific search for the soul, Scribner, New York.

Crick, F & Koch, C 1990, ‘Towards a neurobiological theory of consciousness’, Seminars in the Neurosciences, vol. 2, pp. 263–75.

---- 1998, ‘Consciousness and neuroscience’, Cerebral Cortex, vol. 8, no. 2, pp. 97–107.

Crisinel, A-S & Spence, C 2010, ‘As bitter as a trombone: synesthetic correspondences in nonsynesthetes between tastes/flavors and musical notes’, Attention, Perception, & Psychophysics, vol. 72, no. 7, pp. 1994–2002.

Crosland, A 1927, The Jazz Singer, motion picture, Warner Bros., USA.

Cross, E & Ticini, L 2012, ‘Neuroaesthetics and beyond: new horizons in applying the science of the brain to the art of dance’, Phenomenology and the Cognitive Sciences, vol. 11, no. 1, pp. 5–16.

Crum, PAC & Bregman, AS 2006, ‘Effects of unit formation on the perception of a changing sound’, The Quarterly Journal of Experimental Psychology, vol. 59, no. 3, pp. 543–56.

Cuarón, A 2006, Children of Men, motion picture, USA, UK.

---- 2013a, Gravity "I've Got You" (trailer), online multimedia, viewed 14 June 2017

---- 2013b, Gravity, motion picture, USA.

---- 2018, Roma, motion picture, Mexico, USA.

Cuccio, V 2014, ‘From a bodily-based format of knowledge to symbols. The evolution of human language’, Biosemiotics, vol. 7, no. 1, pp. 49–61.

Cumbow, RC & Johnson, W 1985, ‘Film sound’, Film Quarterly, vol. 39, no. 2, p. 64.

Cummings, EE 1926, ‘since feeling is first’, in AM Eastman, AW Allison, H Barrows, CR Blake, AJ Carr & HM English (eds), The Norton anthology of poetry, W.W. Norton & Company, New York, pp. 1043–4.

Cunningham, C 1996, Second Bad Vilbel, music video, UK.

---- 1999, Engine, television commercial, UK.

Currie, G 1995, Image and mind: film, philosophy and cognitive science, Cambridge University Press, New York, NY.

Curtis, R 2003, Love Actually, motion picture, UK, USA, France.

Cytowic, RE 2002, Synesthesia: a union of the senses, 2nd edn, MIT Press, Cambridge, Massachusetts.

Czarniawska-Joerges, B 2004, ‘The ‘narrative turn’ in social studies’, in Narratives in social science research, SAGE, London, pp. 1–16.

D’Aloia, A 2012, ‘The intangible ground: a neurophenomenology of the film experience’, NECSUS, vol. 1, no. 2, pp. 219–39.

D’Ausilio, A 2007, ‘The role of the mirror system in mapping complex sounds into actions’, The Journal of Neuroscience, vol. 27, no. 22, pp. 5847–8.

Damásio, AR 1989a, ‘Time-locked multiregional retroactivation: a systems-level proposal for the neural substrates of recall and recognition’, Cognition, vol. 33, no. 1/2, pp. 25–62.

---- 1989b, ‘The brain binds entities and events by multiregional activation from convergence zones’, Neural Computation, vol. 1, no. 1, pp. 123–32.

---- 1994, Descartes' error: emotion, reason, and the human brain, Quill, New York.

---- 1995, ‘Toward a neurobiology of emotion and feeling: operational concepts and hypotheses’, The Neuroscientist, vol. 1, no. 1, pp. 19–25.

---- 1996, ‘The somatic marker hypothesis and the possible functions of the prefrontal cortex’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 351, no. 1346, pp. 1413–20.

---- 1998, ‘Investigating the biology of consciousness’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 353, no. 1377, pp. 1879–82.

---- 1999, The feeling of what happens: body, emotion and the making of consciousness, Harcourt Brace, New York.

---- 2000, ‘A second chance for emotion’, in RD Lane, L Nadel & G Ahern (eds), Cognitive neuroscience of emotion, Oxford University Press, New York ; Oxford, pp. 12–23.

---- 2001, ‘Reflections on the neurobiology of emotion and feeling’, in J Branquinh (ed.), The foundations of cognitive science, Clarendon Press, Oxford, pp. 99–108.

---- 2003, Looking for Spinoza: joy, sorrow, and the feeling brain, Vintage, London.

---- 2010, Self comes to mind: constructing the conscious brain, Vintage Books, New York.

Damásio, AR & Meyer, K 2009, ‘Consciousness: an overview of the phenomenon and of its possible neural basis’, in S Laureys & G Tononi (eds), The neurology of consciousness: cognitive neuroscience and neuropathology, Academic, Amsterdam; London, pp. 3–14.

Damásio, AR, Tranel, D & Damásio, H 1991, ‘Somatic markers and the guidance of behavior: theory and preliminary testing’, in HS Levin, HM Eisenberg & AL Benton (eds), Frontal lobe function and dysfunction, Oxford University Press, New York, pp. 217–29.

Damásio, H, Grabowski, T, Frank, R, Galaburda, AM & Damásio, AR 1994, ‘The return of Phineas Gage: clues about the brain from the skull of a famous patient’, Science, vol. 264, no. 5162, pp. 1102–5.

Darwin, C 1872, The expression of the emotions in man and animals, John Murray, London. de Gelder, B 2005, ‘Nonconscious emotions: new findings and perspectives on nonconscious facial expression recognition and its voice and whole-body contexts’, in LF Barrett, PM Niedenthal & P Winkielman (eds), Emotion and consciousness, The Guilford Press, New York; London, pp. 123–49.

304 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. de Gelder, B & Vroomen, J 2000, ‘The perception of emotions by ear and by eye’, Cognition and Emotion, vol. 14, no. 3, pp. 289–311. de Gelder, B & Tamietto, M 2011, ‘A unified science of the non-conscious mind?’, Nature Reviews Neuroscience, vol. 12, no. 5, p. 302. de Gelder, B, van Honk, J & Tamietto, M 2011, ‘Emotion in the brain: of low roads, high roads and roads less travelled’, Nature Reviews Neuroscience, vol. 12, no. 7, p. 425. de Waal, FBM & Ferrari, PF 2010, ‘Towards a bottom-up perspective on animal and human cognition’, Trends in Cognitive Sciences, vol. 14, no. 5, pp. 201–7. de Wied, M 1995, ‘The role of temporal expectancies in the production of film suspense’, Poetics, vol. 23, no. 1- 2, pp. 107–23.

Deadwood, 2004–2006, television drama, Home Box Office, USA.

Debord, G 1967, Society of the spectacle, 1 vols., Black & Red, Detroit.

Decety, J & Jackson, PL 2004, ‘The functional architecture of human empathy’, Behavioral and Cognitive Neuroscience Reviews, vol. 3, no. 2, pp. 71–100.

Decety, J & Lamm, C 2009, ‘The biological basis of empathy and intersubjectivity’, in JT Cacioppo & GG Berntson (eds), Handbook of neuroscience for the behavioral sciences, John Wiley and Sons, New York, pp. 940–57.

Dehaene, S & Naccache, L 2001, ‘Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework’, Cognition, vol. 79, no. 1, pp. 1–37.

Dehaene, S & Changeux, J-P 2004, ‘Neural mechanisms for access to consciousness’, in MS Gazzaniga (ed.), The cognitive neurosciences, 3rd edn, MIT Press, Cambridge, MA, pp. 1145–58.

Dehaene, S, Changeux, J-P, Naccache, L, Sackur, J & Sergent, C 2006, ‘Conscious, preconscious, and subliminal processing: a testable taxonomy’, Trends in Cognitive Sciences, vol. 10, no. 5, pp. 204–11.

Deigh, J 2014, ‘William James and the rise of the scientific study of emotion’, Emotion Review, vol. 6, no. 1, pp. 4–12. del Rìo, E 1996, ‘The body as foundation of the screen: allegories of technology in Atom Egoyan's Speaking Parts’, Camera Obscura, vol. 13, no. 2, pp. 92-115.

---- 2008, Deleuze and the cinemas of performance: powers of affection, Edinburgh University Press, Edinburgh.

Deleuze, G 1986, Cinéma 1: the Movement-Image, Athlone, London.

---- 1989, Cinéma 2: the time-image, Athlone, London.

Di Dio, C & Gallese, V 2009, ‘Neuroaesthetics: a review’, Current Opinion in Neurobiology, vol. 19, no. 6, pp. 682–7.

Di Pellegrino, G, Fadiga, L, Fogassi, L, Gallese, V & Rizzolatti, G 1992, ‘Understanding motor events: a neurophysiological study’, Experimental Brain Research, vol. 91, no. 1, pp. 176–80.

Dickson, WKL 1891, Dickson Greeting, motion picture, USA.

---- 1894/1895, Dickson Experimental Sound Film motion picture, USA.

Dijksterhuis, A & Nordgren, LF 2006, ‘A theory of unconscious thought’, Perspectives on Psychological Science, vol. 1, no. 2, pp. 95–109.

Dijksterhuis, A, Aarts, H & Smith, PK 2005, ‘The power of the subliminal: on subliminal persuasion and other potential applications’, in RR Hassin, JS Uleman & JA Bargh (eds), The new unconscious, Oxford University Press, Oxford ; New York, pp. 77–106.

Doane, MA 1980a, ‘The voice in the cinema: the articulation of body and space’, Yale French Studies, no. 60, pp. 33–50.

---- 1980b, ‘Ideology and the practice of sound editing and mixing’, in T De Lauretis & S Heath (eds), The cinematic apparatus, St. Martin's Press, New York, pp. 47–56.

---- 1985, ‘Ideology and the practice of sound editing and mixing’, in E Weis & J Belton (eds), Film sound: theory and practice, Columbia University Press, New York, pp. 54–62.

Docter, P, Silverman, D & Unkrich, L 2001, Monsters, Inc., motion picture, USA.

Dolby Laboratories 2012, Content creators, distributors, exhibitors: introducing Dolby Atmos, viewed 11 February 2014, .

Domitor, viewed 20 October 2014, .

Donaldson, LF 2014, ‘The work of an invisible body: the contribution of Foley artists to on-screen effort’, Alphaville: Journal of Film and Screen Media, no. 7, pp. 1–15.

---- 2017a, ‘Feeling and filmmaking: the design and affect of film sound’, The New Soundtrack, vol. 7, no. 1, pp. 31–46.

---- 2017b, ‘"You have to feel a sound for it to be effective”: sonic surfaces in film and television’, in M Mera, R Sadoff & B Winters (eds), The Routledge companion to screen music and sound, Routledge, New York, NY ; London, pp. 85–95.

Donnelly, KJ 2005, The spectre of sound: music in film and television, BFI, London.

---- 2014, Occult aesthetics: synchronization in sound film, Oxford University Press, New York.

Droit-Volet, S & Meck, WH 2007, ‘How emotions colour our perception of time’, Trends in Cognitive Sciences, vol. 11, no. 12, pp. 504–13.

Droit-Volet, S & Gil, S 2009, ‘The time-emotion paradox’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, no. 1525, pp. 1943–53.

Droit-Volet, S, Fayolle, SL & Gil, S 2011, ‘Emotion and time perception: effects of film-induced mood’, Frontiers in Integrative Neuroscience, vol. 5, Article 33, pp. 1–9, doi 10.3389/fnint.2011.00033, viewed 13 December 2011, .

Dudai, Y 2012, ‘The cinema-cognition dialogue: a match made in brain’, Frontiers in Human Neuroscience, vol. 6, Article 248, pp. 1–8, doi 10.3389/fnhum.2012.00248, viewed 17 September 2014, .

Dunbar, RIM 1992, ‘Neocortex size as a constraint on group size in primates’, Journal of Human Evolution, vol. 22, no. 6, pp. 469–93.

---- 1993, ‘Coevolution of neocortical size, group size and language in humans’, Behavioral and Brain Sciences, vol. 16, no. 4, pp. 681–94.

---- 1998, ‘The social brain hypothesis’, Evolutionary Anthropology, vol. 6, no. 5, pp. 178–90.

Duque, JFD, Turner, R, Lewis, ED & Egan, G 2010, ‘Neuroanthropology: a humanistic science for the study of the culture-brain nexus’, Social Cognitive and Affective Neuroscience, vol. 5, no. 2-3, pp. 138–47.

306 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Durgnat, R 1980, ‘The death of cinesemiology (with not even a whimper)’, Cinéaste, vol. 10, no. 2, pp. 10–3.

---- 1982, ‘The language of film and problems in cinesemiotics’, The British Journal of Aesthetics, vol. 22, no. 3, pp. 264–9.

Dyson, F 2009, Sounding new media: immersion and embodiment in the arts and culture, University of California Press, Berkeley.

Edelman, GM 1987, Neural Darwinism: the theory of neuronal group selection, Basic Books, New York, NY.

---- 1989, The remembered present: a biological theory of consciousness, Basic Books, New York, NY.

---- 1992, Bright air, brilliant fire: on the matter of the mind, Basic Books, New York, NY.

---- 2003, ‘Naturalizing consciousness: a theoretical framework’, Proceedings of the National Academy of Sciences, vol. 100, no. 9, pp. 5520–4.

---- 2004, Wider than the sky: the phenomenal gift of consciousness, Yale University Press, New Haven.

Edelman, GM & Mountcastle, VB 1978, The mindful brain: cortical organization and the group-selective theory of higher brain function, Neurosciences Research Program, MIT Press, Cambridge.

Edelman, GM & Tononi, G 2001, Consciousness: how matter becomes imagination, Penguin, London.

Eibl, K 2012, ‘How can evolutionary biology enrich the study of literature?’, in C Gansel & D Vanderbeke (eds), Telling stories: literature and evolution, De Gruyter, Berlin; Boston, pp. 11–29.

Eisenstein, SM 1925, Stachka, Soviet Union.

---- 1947 [1923], ‘Montage of attractions, an essay’, in The film sense, Harcourt, Brace & World, New York, pp. 229–33.

Eisenstein, SM, Pudovkin, VI & Aleksandrov, GV 1928, ‘The sound film: a statement from the USSR’, Close Up, no. 3/4 October, pp. 10–3.

Eitan, Z 2013, ‘How pitch and loudness shape musical space and motion: new findings and persisting questions’, in S-L Tan, A Cohen, SD Lipscomb & RA Kendall (eds), The psychology of music in multimedia, Oxford University Press, Oxford, UK, pp. 161–87.

Eitan, Z & Rothschild, I 2011, ‘How music touches: musical parameters and listeners’ audio-tactile metaphorical mappings’, Psychology of Music, vol. 39, no. 4, pp. 449–67.

Eitan, Z, Schupak, A & Marks, LE 2008, Louder is higher: cross-modal interaction of loudness change and vertical motion in speeded classification, paper presented at 10th International Conference on Music Perception and Cognition, Sapporo, Japan, 25–29 August 2008.

Eitan, Z, Schupak, A, Gotler, A & Marks, LE 2014, ‘Lower pitch is larger, yet falling pitches shrink: interaction of pitch change and size change in speeded discrimination’, Experimental Psychology, vol. 61, no. 4, pp. 273–84.

Eliot, TS 1942, ‘Little Gidding’, in The Poems of T. S. Eliot, Faber & Faber, London, vol. 1, pp. 201–9.

Elliott, P 2011, Hitchcock and the cinema of sensations: embodied film theory and cinematic reception, International library of visual culture v 2, I.B. Tauris, London ; New York.

Ellis, RJ & Simons, RF 2005, ‘The impact of music on subjective and physiological indices of emotion while viewing films’, Psychomusicology, vol. 19, no. 1, pp. 15–40.

Elsaesser, T 1998, ‘Specularity and engulfment: Francis Ford Coppola and Bram Stoker's Dracula’, in S Neale & M Smith (eds), Contemporary Hollywood cinema, Routledge, London; New York, pp. 191–208.

---- 2002, ‘Classical / post-classical narrative’, in T Elsaesser & W Buckland (eds), Studying contemporary American film: a guide to movie analysis, Arnold, London, pp. 26–79.

Elsaesser, T & Hagener, M 2010a, Film theory: an introduction through the senses, Routledge, New York.

---- 2010b, ‘Film theory, cinema, the body and the senses’, in Film theory: an introduction through the senses, Routledge, New York, pp. 1–12.

Elvemo, J-M 2013, ‘Spatial perception and diegesis in multi-channel surround cinema’, The New Soundtrack, vol. 3, no. 1, pp. 31–44.

Erlmann, V (ed.) 2004, Hearing cultures: essays on sound, listening, and modernity, Wenner-Gren Center international symposium series., Berg, Oxford ; New York, NY.

Escoffier, N, Sheng, DYJ & Schirmer, A 2010, ‘Unattended musical beats enhance visual processing’, Acta Psychologica, vol. 135, no. 1, pp. 12–6.

Escoffier, N, Herrmann, CS & Schirmer, A 2015, ‘Auditory rhythms entrain visual processes in the human brain: evidence from evoked oscillations and event-related potentials’, NeuroImage, vol. 111, pp. 267– 76.

Etzi, R, Spence, C, Zampini, M & Gallace, A 2016, ‘When sandpaper is ‘kiki’ and satin is ‘bouba’: an exploration of the associations between words, emotional states, and the tactile attributes of everyday materials’, Multisensory Research, vol. 29, no. 1-3, pp. 133–55.

Eyman, S 1999, The speed of sound: Hollywood and the talkie revolution, 1926-1930, Johns Hopkins University Press, Baltimore.

Fahlenbrach, K 2002, Feeling sounds. Emotional aspects of music videos, paper presented at 8th Conference of the International Society for the Empirical Study of Literature and Media, Pécs, Hungary, August 2002.

---- 2005a, ‘The emotional design of music videos. Approaches to audiovisual metaphors’, Journal of Moving Image Studies, vol. 3, no. 1, pp. 22–8.

---- 2005b, ‘Aesthetics and audiovisual metaphors in media perception’, CLCWeb: Comparative Literature and Culture, vol. 7, no. 4, Article 3, doi 10.7771/1481-4374.1280, viewed 2 August 2012, .

---- 2007, ‘Embodied spaces: film spaces as (leading) audiovisual metaphors’, in JD Anderson & BF Anderson (eds), Narration and spectatorship in moving images, Cambridge Scholars Publishing, Newcastle, pp. 105–21.

---- 2008, ‘Emotions in sound: audiovisual metaphors in the sound design of narrative films’, Projections, vol. 2, no. 2, pp. 85–103.

---- 2016a, ‘Audiovisual metaphors and embodiment in film, television, and video games: a comparative discussion’, in LC Grabbe, P Rupert-Kruse & NM Schmitz (eds), Image embodiment: new perspectives of the sensory turn, Büchner-Verlag, Darmstadt, pp. 135–60.

---- 2016b, ‘Audiovisual metaphors as embodied narratives in moving images’, in K Fahlenbrach (ed.), Embodied metaphors in film, television, and video games: cognitive approaches, Routledge, New York ; London, pp. 33–50.

---- 2017, ‘Sonic spaces in movies: audiovisual metaphors and embodied meanings in sound design’, in C Wöllner (ed.), Body, sound and space in music and beyond: multimodal explorations, Routledge, Abingdon, Oxon; New York, NY, pp. 129–49.

308 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Fairfax, D 2017, ‘The audio-spectator: an interview with Michel Chion’, Senses of Cinema, no. 84, Article viewed 30 September 2018, .

Farley, S 2008, Designing Sound, website, viewed 18 January 2019

Fauconnier, G 1985, Mental spaces: aspects of meaning construction in natural language, MIT Press, Cambridge, MA.

---- 2005, ‘Compression and emergent structure’, Language and Linguistics, vol. 6, no. 4, pp. 523–38.

Fauconnier, G & Turner, M 2002, The way we think: conceptual blending and the mind's hidden complexities, Basic Books, New York.

Fehr, B & Russell, JA 1994, ‘Fuzzy concepts in a fuzzy hierarchy: varieties of anger’, Journal of Personality and Social Psychology, vol. 67, no. 2, pp. 186–205.

Feigl, H 1958, ‘The “mental” and the “physical”’, in H Feigl, M Scriven & G Maxwell (eds), Minnesota Studies in the Philosophy of Science, University of Minnesota Press, Minneapolis, vol. 2, pp. 370–497.

Fendrich, R & Corballis, PM 2001, ‘The temporal cross-capture of audition and vision’, Perception and Psychophysics, vol. 63, no. 4, pp. 719–25.

Fiala, B, Arico, A & Nichols, S 2011, ‘On the psychological origins of dualism: dual-process cognition and the explanatory gap’, in E Slingerland & M Collard (eds), Creating consilience: integrating the Sciences and the Humanities, Oxford University Press, New York, pp. 88–110.

Figgis, M 2000, Timecode, motion picture, USA.

Fincher, D 2011, The Girl with the Dragon Tattoo (teaser), online multimedia, viewed 10 May 2015

Finney, EM, Fine, I & Dobkins, KR 2001, ‘Visual stimuli activate auditory cortex in the deaf’, Nature Neuroscience, vol. 4, no. 12, pp. 1171–3.

Finney, EM, Clementz, BA, Hickok, G & Dobkins, KR 2003, ‘Visual stimuli activate auditory cortex in deaf subjects: evidence from MEG’, NeuroReport, vol. 14, no. 11, pp. 1425–7.

Fischer, KW, Shaver, PR & Carnochan, P 1990, ‘How emotions develop and how they organise development’, Cognition and Emotion, vol. 4, no. 2, pp. 81–127.

Fischer, L 1977/1978, ‘Enthusiasm: from kino-eye to radio-eye’, Film Quarterly, vol. 31, no. 2, pp. 25–34.

---- 2003, ‘Sunrise: a song of two humans’, in R White & E Buscombe (eds), British Film Institute Film Classics, BFI/Fitzroy Dearborn, London ; New York, vol. 1, pp. 72–93.

Fitzgerald, D & Callard, F 2015, ‘Social science and neuroscience beyond interdisciplinarity: experimental entanglements’, Theory, Culture & Society, vol. 32, no. 1, pp. 3–32.

Flaxman, G 2000, The brain is the screen: Deleuze and the philosophy of cinema, University of Minnesota Press, Minneapolis.

Fleming, V 1939, The Wizard of Oz, motion picture, USA.

Flinn, C 1992, Strains of Utopia : gender, nostalgia, and Hollywood film music, Princeton University Press, Princeton, N.J.

Flueckiger, B 2007, ‘Sound effects. Strategies for sound effects in film’, in G Harper, R Doughty & J Eisentraut (eds), Sound and music in film and visual media: a critical overview, Continuum, London, pp. 151–79.

309 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Fogassi, L 2011, ‘The mirror neuron system: how cognitive functions emerge from motor organization’, Journal of Economic Behavior & Organization, vol. 77, no. 1, pp. 66–75.

Fogassi, L, Ferrari, PF, Gesierich, B, Rozzi, S, Chersi, F & Rizzolatti, G 2005, ‘Parietal lobe: from action organization to intention understanding’, Science, vol. 308, no. 5722, pp. 662–7.

Forde, T 2011, ‘The Sunshine soundtrack as aural attraction’, New Review of Film and Television Studies, vol. 9, no. 1, pp. 71 – 83.

Fourneret, P & Jeannerod, M 1998, ‘Limited conscious monitoring of motor performance in normal subjects’, Neuropsychologia, vol. 36, no. 11, pp. 1133–40.

Fox, J 1980, ‘Walter Murch: making beaches out of grains of sand’, Cinefex, no. 3, pp. 42–57.

Frassinetti, F, Bolognini, N & Làdavas, E 2002, ‘Enhancement of visual perception by crossmodal visuo- auditory interaction’, Experimental Brain Research, vol. 147, no. 3, pp. 332–43.

Freedberg, D & Gallese, V 2007, ‘Motion, emotion and empathy in esthetic experience’, Trends in Cognitive Sciences, vol. 11, no. 5, pp. 197–203.

Freeman, E & Driver, J 2008, ‘Direction of visual apparent motion driven solely by timing of a static sound’, Current Biology, vol. 18, no. 16, pp. 1262–6.

Freytag, G 1863, Die Technik des Dramas, Kessinger Publishing.

Friedkin, W 1971, The French Connection, motion picture, USA.

Frijda, NH 1984, ‘Toward a model of emotion’, in CD Spielberger, PB Defares & IG Sarason (eds), Stress and anxiety, Hemisphere Press, New York, vol. 9, pp. 3–16.

---- 1986, The emotions, Studies in emotion and social interaction., Cambridge University Press, Cambridge ; New York.

---- 1988, ‘The laws of emotion’, American Psychologist, vol. 43, no. 5, pp. 349–58.

---- 2004, ‘The psychologist's point of view’, in M Lewis & JM Haviland-Jones (eds), Handbook of emotions, 2nd edn, Guilford Press, New York, pp. 91–115.

Frijda, NH, Mesquita, B, Sonnemans, J & van Goozen, S 1991, ‘The duration of affective phenomena, or emotions, sentiments and passions’, in KT Strongman (ed.), International review of studies on emotion, John Wiley & Sons, Chichester, England ; New York, N.Y., pp. 187–226.

Frith, CD 2005, ‘The quest for consciousness: a neurobiological approach (review)’, The American Journal of Psychiatry, vol. 162, no. 2, p. 407.

---- 2007, ‘The social brain?’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 362, no. 1480, pp. 671–8.

Frith, CD 2010a, ‘What is consciousness for?’, Pragmatics and Cognition, vol. 18, no. 3, pp. 497–551.

---- 2010b, What is consciousness for?, Collège de France, viewed 12 February 2015, .

Frith, CD & Frith, U 1999, ‘Interacting minds - a biological basis’, Science, vol. 286, no. 5445, pp. 1692–5.

---- 2005, ‘Theory of mind’, Current Biology, vol. 15, no. 17, pp. R644–R5.

---- 2007, ‘Social cognition in humans’, Current Biology, vol. 17, no. 16, pp. R724–R32.

---- 2012, ‘Mechanisms of social cognition’, Annual Review of Psychology, vol. 63, no. 1, pp. 287–313.

Frith, U & Frith, CD 2001, ‘The biological basis of social interaction’, Current Directions in Psychological Science, vol. 10, no. 5, pp. 151–5.

---- 2010, ‘The social brain: allowing humans to boldly go where no other species has been’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 365, no. 1537, pp. 165–76.

Gabrielsson, A 2001, ‘Emotions in strong experiences with music’, in PN Juslin & JA Sloboda (eds), Music and emotion: theory and research, Oxford University Press, New York, NY, pp. 431–49.

Galati, G, Committeri, G, Spitoni, G, Aprile, T, Di Russo, F, Pitzalis, S & Pizzamiglio, L 2008, ‘A selective representation of the meaning of actions in the auditory mirror system’, NeuroImage, vol. 40, no. 3, pp. 1274–86.

Gallagher, S 2005, How the body shapes the mind, Oxford University Press, Oxford.

---- 2015, ‘Invasion of the body snatchers: how embodied cognition is being disembodied’, The Philosophers’ Magazine, no. 2, pp. 96–102.

Gallese, V 2001, ‘The 'shared manifold' hypothesis: from mirror neurons to empathy’, Journal of Consciousness Studies, vol. 8, no. 5-7, pp. 33–50.

---- 2003a, ‘The manifold nature of interpersonal relations: the quest for a common mechanism’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 358, no. 1431, pp. 517–28.

---- 2003b, ‘The roots of empathy: the shared manifold hypothesis and the neural basis of intersubjectivity’, Psychopathology, vol. 36, no. 4, pp. 171–80.

---- 2005, ‘Embodied simulation: from neurons to phenomenal experience’, Phenomenology and the Cognitive Sciences, vol. 4, no. 1, pp. 23–48.

---- 2006, ‘Embodied simulation: from mirror neuron systems to interpersonal relations’, in Empathy and Fairness, John Wiley & Sons, Chichester, UK, vol. 278, pp. 3–19.

---- 2007a, ‘Before and below ‘theory of mind’: embodied simulation and the neural correlates of social cognition’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 362, no. 1480, pp. 659–69.

---- 2007b, ‘The “conscious” dorsal stream: embodied simulation and its role in space and action conscious awareness’, Psyche, vol. 13, no. 1, pp. 1–20.

---- 2008, ‘Mirror neurons and the social nature of language: the neural exploitation hypothesis’, Social Neuroscience, vol. 3, no. 3, pp. 317–33.

---- 2009, ‘Mirror neurons, embodied simulation, and the neural basis of social identification’, Psychoanalytic Dialogues, vol. 19, no. 5, pp. 519–36.

---- 2011a, ‘Neuroscience and phenomenology’, Phenomenology and Mind, vol. 1, pp. 33–48.

---- 2011b, ‘Embodied simulation theory: imagination and narrative’, Neuropsychoanalysis, vol. 13, no. 2, pp. 196–200.

---- 2011c, ‘Seeing art ... beyond vision. Liberated embodied simulation in aesthetic experience’, in A Abbushi, I Franke & I Mommenejad (eds), Seeing with the eyes closed, Association for Neuroesthetics, Berlin, pp. 62–5.

---- 2013, ‘Mirror neurons, embodied simulation and a second-person approach to mindreading’, Cortex, vol. 49, no. 10, pp. 2954–6.

311 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Gallese, V & Goldman, AI 1998, ‘Mirror neurons and the simulation theory of mind-reading’, Trends in Cognitive Sciences, vol. 2, no. 12, pp. 493–501.

Gallese, V & Lakoff, G 2005, ‘The brain’s concepts: the role of the sensory-motor system in conceptual knowledge’, Cognitive Neuropsychology, vol. 22, no. 3, pp. 455–79.

Gallese, V & Freedberg, D 2007, ‘Mirror and canonical neurons are crucial elements in esthetic response’, Trends in Cognitive Sciences, vol. 11, no. 10, p. 411.

Gallese, V & Sinigaglia, C 2011a, ‘What is so special about embodied simulation?’, Trends in Cognitive Sciences, vol. 15, no. 11, pp. 512–9.

---- 2011b, ‘How the body in action shapes the self’, Journal of Consciousness Studies, vol. 18, no. 7-8, pp. 117– 43.

Gallese, V & Guerra, M 2012a, ‘Embodying movies: Embodied Simulation and Film Studies’, Cinema: Journal of Philosophy and the Moving Image, vol. 3, pp. 183–210.

---- 2012b, ‘Embodied Simulation and Film Studies’, in PSC Branco, SrD Branco & S Viegas (eds), Embodiment and the body Instituto de Filosofia da Linguagem, Lisbon, Portugal, vol. 3, pp. 183–210.

Gallese, V & Di Dio, C 2012, ‘Neuroesthetics: the body in esthetic experience’, in VS Ramachandran (ed.), Encyclopedia of human behavior, 2nd edn, Elsevier Academic Press, Amsterdam, pp. 687–93.

Gallese, V & Guerra, M 2014, ‘The feeling of motion: camera movements and motor cognition’, Cinéma & Cie, vol. 14, no. 22/23, pp. 103–12.

Gallese, V & Cuccio, V 2015, ‘The paradigmatic body. Embodied simulation, intersubjectivity, the bodily self, and language’, in T Metzinger & JM Windt (eds), OpenMIND, MIND Group, Frankfurt am Main, pp. 1–5.

Gallese, V, Eagle, M & Migone, P 2007, ‘Intentional attunement: mirror neurons and the neural underpinnings of interpersonal relations’, Journal of the American Psychoanalytic Association, vol. 55, no. 1, pp. 131– 75.

Game of Thrones, 2011– television drama, Home Box Office, USA, UK.

Gärdenfors, P 2003, How homo became sapiens: on the evolution of thinking, Oxford University Press, Oxford.

---- 2007, ‘Cognitive semantics and image schemas with embodied forces’, in JM Krois, M Rosengren, A Steidele & D Westerkamp (eds), Embodiment in cognition and culture, John Benjamins Publishing Company, Amsterdam ; Philadelphia, pp. 57–76.

Garner, T & Grimshaw, M 2014, ‘Sonic virtuality: understanding audio in a virtual world’, in M Grimshaw (ed.), The Oxford Handbook of Virtuality, Oxford University Press, New York, pp. 364–77.

Garner, TA 2018, Echoes of other worlds: sound in virtual reality, Palgrave Macmillan, Cham, Switzerland.

Gaudreault, A 1983, ‘Temporality and narrativity in Early Cinema, 1895-1908’, Film before Griffith, p. 311ñ29.

---- 1987, ‘Narration and monstration in the cinema’, Journal of Film and Video, vol. 39, no. 2, pp. 29–36.

---- 2006, ‘From "Primitive Cinema" to "Kine-Attractography"’, in W Strauven (ed.), The cinema of attractions reloaded: film culture in transition, Amsterdam University Press, Amsterdam, pp. 85–104.

---- 2009, From Plato to Lumière: narration and monstration in literature and cinema, University of Toronto Press, Toronto.

312 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Gaudreault, A & Gunning, T 1989, ‘Le cinéma des premier temps: un défi a histoire du film?’, in J Aumont, A Gaudreault & M Marie (eds), Histoire du cinéma: nouvelles approches, Publications de la Sorbonne, Paris, pp. 49–63.

Gaver, WW 1986, ‘Auditory icons: using sound in computer interfaces’, Human-Computer Interaction, vol. 2, no. 2, pp. 167–77.

---- 1988, ‘Everyday listening and auditory icons’, Doctor of Philosophy, University of California, San Diego.

---- 1993a, ‘How do we hear in the world? Explorations in ecological acoustics’, Ecological Psychology, vol. 5, no. 4, pp. 285–313.

---- 1993b, ‘What in the world do we hear? An ecological approach to auditory event perception’, Ecological Psychology, vol. 5, no. 1, pp. 1–29.

Gazzaniga, MS 1998, ‘Brain and conscious experience’, in H Jasper, L Descarries, V Castellucci & S Rossignol (eds), Consciousness: at the frontiers of neuroscience, Lippincott-Raven, Philadelphia.

Gazzola, V, Aziz-Zadeh, L & Keysers, C 2006, ‘Empathy and the somatotopic auditory mirror system in humans’, Current Biology, vol. 16, no. 18, pp. 1824–9.

Geen, RG & Quanty, MB 1977, ‘The catharsis of aggression: an evaluation of a hypothesis’, in L Berkowitz (ed.), Advances in experimental social psychology, Academic Press, New York, vol. 10, pp. 1–37.

Gelder, K 1999, ‘Jane Campion and the limits of literary cinema’, in D Cartmell & I Whelehan (eds), Adaptations: from text to screen, screen to text, Routledge, New York, pp. 157–71.

Gendron, M 2010, ‘Defining emotion: a brief history’, Emotion Review, vol. 2, no. 4, pp. 371–2.

Gendron, M & Barrett, LF 2009, ‘Reconstructing the past: a century of ideas about emotion in psychology’, Emotion Review, vol. 1, no. 4, pp. 316–39.

Gerdes, ABM, Wieser, MJ & Alpers, GW 2014, ‘Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains’, Frontiers in Psychology, vol. 5, Article 1351, pp. 1– 13, doi 10.3389/fpsyg.2014.01351, viewed 17 March 2015, .

Gerdes, ABM, Wieser, MJ, Bublatzky, F, Kusay, A, Plichta, MM & Alpers, GW 2013, ‘Emotional sounds modulate early neural processing of emotional pictures’, Frontiers in Psychology, vol. 4, Article 741, pp. 1–12, doi 10.3389/fpsyg.2013.00741, viewed 23 February 2014, .

Gerrig, RJ 1993, Experiencing narrative worlds: on the psychological activities of reading, Yale University Press, New Haven.

Gerstein, LH, Rountree, C & Ordonez, A 2007, ‘An anthropological perspective on multicultural counselling’, Counselling Psychology Quarterly, vol. 20, no. 4, pp. 375–400.

Getz, I & Lubart, TI 1998, ‘The emotional resonance model of creativity: theoretical and practical extensions’, in SW Russ (ed.), Affect, creative experience, and psychological adjustment, Brunner/Mazel, Philadelphia, pp. 41–56.

Getzmann, S 2007, ‘The effect of brief auditory stimuli on visual apparent motion’, Perception, vol. 36, no. 7, pp. 1089–103.

Geurts, KL 2005, ‘Consciousness as ‘feeling in the body’. A West Arican theory of embodiment, emotion and the making of mind’, in D Howes (ed.), Empire of the senses: the sensual culture reader, Berg, Oxford ; New York, pp. 164–78.

313 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Ghazanfar, AA & Schroeder, CE 2006, ‘Is neocortex essentially multisensory?’, Trends in Cognitive Sciences, vol. 10, no. 6, pp. 278–85.

Gibson, JJ 1966, The senses considered as perceptual systems, Houghton Mifflin, Boston.

---- 1977, ‘The theory of affordances’, in R Shaw & J Bransford (eds), Perceiving, acting, and knowing: toward an ecological psychology, Lawrence Erlbaum, Hillsdale, NJ, pp. 67–82.

Giovannoni, D, Feaster, P, Martin, R & Hennessey, M 2008, First Sounds: Édouard-Léon Scott de Martinville's phonautograms, viewed 3 February 2009, .

Glenberg, AM 1997, ‘What memory is for’, Behavioral and Brain Sciences, vol. 20, no. 1, pp. 41–50.

---- 1999, ‘Why mental models must be embodied’, in R Gert & H Christopher (eds), Advances in Psychology, North-Holland, Amsterdam, vol. Volume 128, pp. 77–90.

Glenberg, AM & Robertson, DA 2000, ‘Symbol grounding and meaning: a comparison of high-dimensional and embodied theories of meaning’, Journal of Memory and Language, vol. 43, no. 3, pp. 379–401.

‘Global experiments in early synchronous sound’, 1999, Film History, vol. 11, no. 4, pp. 404–7.

Goldman, AI 1992, ‘In defense of the Simulation Theory’, Mind & Language, vol. 7, no. 1-2, pp. 104–19.

---- 2014, ‘The bodily formats approach to embodied cognition’, in U Kriegel (ed.), Current controversies in philosophy of mind, Routledge, New York; London, pp. 91–108.

Goldman, AI & de Vignemont, F 2009, ‘Is social cognition embodied?’, Trends in Cognitive Sciences, vol. 13, no. 4, pp. 154–9.

Goldstone, RL & Barsalou, LW 1998, ‘Reuniting perception and conception’, Cognition, vol. 65, no. 2-3, pp. 231–62.

Gondry, M 1997, Around the World, music video, France.

---- 2001a, Making of Star Guitar, online multimedia, viewed 13 June 2011

---- 2001b, Star Guitar, music video, France.

---- 2002, Come into My World, music video, France.

Gopnik, B 2011, ‘Aesthetic science and artistic knowledge’, in AP Shimamura & SE Palmer (eds), Aesthetic science: connecting minds, brains, and experience, Oxford University Press, Oxford, pp. 129–59.

Gorbman, C 1987, Unheard melodies: narrative film music, Indiana University Press, Bloomington.

Goydke, KN, Altenmüller, E, Möller, J & Münte, TF 2004, ‘Changes in emotional tone and instrumental timbre are reflected by the mismatch negativity’, Cognitive Brain Research, vol. 21, no. 3, pp. 351–9.

Grahn, JA, Henry, MJ & McAuley, JD 2011, ‘fMRI investigation of cross-modal interactions in beat perception: audition primes vision, but not vice versa’, NeuroImage, vol. 54, no. 2, pp. 1231–43.

Gratier, M & Trevarthen, C 2008, ‘Musical narrative and motives for culture in mother–infant vocal interaction’, Journal of Consciousness Studies, vol. 15, no. 10-11, pp. 122–58.

Grau, R 1914, The theatre of science: a volume of progress and achievement in the motion picture industry, Broadway Publishing Company, New York.

Greco, M & Stenner, P 2008, ‘Introduction: emotion and social science’, in M Greco & P Stenner (eds), Emotions: a social science reader, Routledge, London ; New York, pp. 1–21.

Green, MC & Brock, TC 2000, ‘The role of transportation in the persuasiveness of public narratives’, Journal of Personality and Social Psychology, vol. 79, no. 5, pp. 701–21.

Green, MC & Donahue, JK 2009, ‘Simulated worlds: transportation into narratives’, in KD Markman, WM Klein & JA Suhr (eds), Handbook of imagination and mental simulation, Psychology Press, New York, pp. 241–56.

Greene, L & Kulezic-Wilson, D (eds) 2016, The Palgrave handbook of sound design and music in screen media, Palgrave Macmillan, London.

Greimas, A 1983, Structural semantics: an attempt at a method, University of Nebraska Press, Lincoln.

Grewe, O, Kopiez, R & Altenmüller, E 2009, ‘Chills as an indicator of individual emotional peaks’, Annals of the New York Academy of Sciences, vol. 1169, no. 1, pp. 351–4.

Grewe, O, Nagel, F, Kopiez, R & Altenmüller, E 2005, ‘How does music arouse “chills”?’, Annals of the New York Academy of Sciences, vol. 1060, no. 1, pp. 446–9.

Grewe, O, Nagel, F, Kopiez, R & Altenmuller, E 2007, ‘Emotions over time: synchronicity and development of subjective, physiological, and facial affective reactions to music’, Emotion, vol. 7, no. 4, pp. 774–88.

Grewe, O, Katzur, B, Kopiez, R & Altenmüller, E 2011, ‘Chills in different sensory domains: frisson elicited by acoustical, visual, tactile and gustatory stimuli’, Psychology of Music, vol. 39, no. 2, pp. 220–39.

Grimshaw, M 2008, The acoustic ecology of the first-person shooter: the player experience of sound in the first- person shooter computer game, VDM Verlag, Saarbrücken, Germany.

Grimshaw, M & Garner, T 2015, Sonic virtuality: sound as emergent perception, Oxford University Press, Oxford; New York.

Grindon, L 1994, ‘The role of spectacle and excess in the critique of illusion’, Post Script, vol. 13, no. 2, pp. 35– 43.

Grodal, TK 1997, Moving pictures: a new theory of film genres, feelings, and cognition, Clarendon Press; Oxford University Press, Oxford; New York.

---- 2006, ‘The PECMA Flow: a general model of visual aesthetics’, Film Studies, vol. 8, no. 1, pp. 1–11.

---- 2009a, Audiovisual salience, PECMA aesthetics and the dopaminergic seeking system, paper presented at Copenhagen Neuroaesthetics Conference, University of Copenhagen, Copenhagen, September 24-26, 2009.

---- 2009b, Embodied visions: evolution, emotion, culture, and film, Oxford University Press, Oxford ; New York.

---- 2009c, ‘The PECMA flow: a general theory of the film experience’, in Embodied visions: evolution, emotion, culture, and film, Oxford University Press, Oxford ; New York, pp. 145–57.

Gross, PR & Levitt, N 1994, Higher superstition: the academic left and its quarrels with science, Johns Hopkins University Press, Baltimore.

Guerra, M 2015, ‘Modes of action at the movies, or re-thinking film style from the embodied perspective’, in P Kravanja & M Coëgnarts (eds), Embodied cognition and cinema, Leuven University Press, Leuven, Belgium, pp. 139–54.

Gumbrecht, HU 2004, Production of presence: what meaning cannot convey, Stanford University Press, Stanford, Calif.

315 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Gunning, T 1986, ‘The cinema of attraction: early film, its spectator and the Avant-Garde’, Wide Angle, vol. 8, no. 3-4, pp. 63–70.

---- 1989, ‘"Primitive" cinema: a frame-up? Or the trick's on us’, Cinema Journal, vol. 28, no. 2, pp. 3–12.

---- 1990, ‘The cinema of attractions: early film, its spectator and the Avant-Garde’, in T Elsaesser & A Barker (eds), Early cinema: space, frame, narrative, BFI Publishing, London, pp. 56–62.

---- 1991, D.W. Griffith and the origins of American narrative film: the early years at Biograph, University of Illinois Press, Chicago.

---- 1993, ‘"Now you see it, now you don't": the temporality of the cinema of attractions’, Velvet Light Trap, vol. 32, no. Fall, pp. 3–12.

---- 1995, ‘An aesthetic of astonishment: early film and the (in)credulous spectator’, in L Williams (ed.), Viewing positions: ways of seeing film, Rutgers University Press, New Brunswick, pp. 114–34.

---- 2001, ‘Doing for the eye what the phonograph does for the ear’, in R Abel & R Altman (eds), The sounds of early cinema, Indiana University Press, Bloomington, IN, pp. 13–31.

Haggis, P 2005, Crash, motion picture, Germany, USA.

Hagood, M 2014, ‘Unpacking a punch: transduction and the sound of combat foley in Fight Club’, Cinema Journal, vol. 53, no. 4, pp. 98–120.

Halberstadt, J, Winkielman, P, Niedenthal, PM & Dalle, N 2009, ‘Emotional conception: how embodied emotion concepts guide perception and facial action’, Psychological Science, vol. 20, no. 10, pp. 1254– 61.

Hall, DA & Moore, DR 2003, ‘Auditory neuroscience: the salience of looming sounds’, Current Biology, vol. 13, no. 3, pp. R91–R3.

Hall, ET 1963, ‘A system for the notation of proxemic behavior’, American Anthropologist, vol. 65, no. 5, pp. 1003–26.

---- 1966, The hidden dimension, Doubleday, Garden City, NY.

Halloran, K 2007, ‘Mathematical and scientific forms of knowledge: a systemic functional multimodal grammatical approach’, in F Christie & JR Martin (eds), Language, knowledge and pedagogy: functional linguistic and sociological perspectives, Continuum, London; New York, NY, pp. 205–36.

Halpern, DL, Blake, R & Hillenbrand, J 1986, ‘Psychoacoustics of a chilling sound’, Perception & Psychophysics, vol. 39, no. 2, pp. 77–80.

Han, S & Northoff, G 2008, ‘Culture-sensitive neural substrates of human cognition: a transcultural neuroimaging approach’, Nature Review Neuroscience, vol. 9, no. 8, pp. 646–54.

---- 2009, ‘Understanding the self: a cultural neuroscience approach’, in YC Joan (ed.), Progress in Brain Research, Elsevier, Amsterdam, vol. 178, pp. 203–12.

Han, S, Northoff, G, Vogeley, K, Wexler, BE, Kitayama, S & Varnum, MEW 2013, ‘A cultural neuroscience approach to the biosocial nature of the human brain’, Annual Review of Psychology, vol. 64, no. 1, pp. 335–59.

Handel, S 1988, ‘Space is to time as vision is to audition: seductive but misleading’, Journal of Experimental Psychology: Human Perception and Performance, vol. 14, no. 2, pp. 315–7.

Hanser, WE & Mark, RE 2013, ‘Music influences ratings of the affect of visual stimuli’, Psihologijske Teme, vol. 22, no. 2, pp. 305–24.

316 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Hanson, H 2007, ‘Sound affects: post-production sound, soundscapes and sound design in Hollywood's studio era’, Music, Sound, and the Moving Image, vol. 1, no. 1, pp. 27–49.

---- 2017, Hollywood soundscapes: film sound style, craft and production in the classical era, Bloomsbury Academic, London.

Harper, G, Doughty, R & Eisentraut, J (eds) 2009, Sound and music in film and visual media: a critical overview, Continuum, London.

Harré, R 1986, ‘An outline of the social constructionist viewpoint’, in R Harré (ed.), The social construction of emotions, Blackwell, Oxford, UK ; New York, NY, pp. 2–14.

Harrison, L & Loui, P 2014, ‘Thrills, chills, frissons, and skin orgasms: toward an integrative model of transcendent psychophysiological experiences in music’, Frontiers in Psychology, vol. 5, Article 790, pp. 1–6, doi 10.3389/fpsyg.2014.00790, viewed 12 September 2014, .

Hatfield, E, Cacioppo, JT & Rapson, RL 1993, ‘Emotional contagion’, Current Directions in Psychological Science, vol. 2, no. 3, pp. 96–100.

---- 1994, Emotional contagion, Cambridge University Press, New York.

Hedger, SC, Nusbaum, HC, Lescop, O, Wallisch, P & Hoeckner, B 2013, ‘Music can elicit a visual motion aftereffect’, Attention, Perception, & Psychophysics, vol. 75, no. 5, pp. 1039–47.

Heeter, C 1992, ‘Being there: the subjective experience of presence’, Presence: Teleoperators and Virtual Environments, vol. 1, no. 2, pp. 262–71.

Heimann, K, Umilta, MA, Guerra, M & Gallese, V 2014, ‘Moving mirrors: a high-density EEG study investigating the effect of camera movements on motor cortex activation during action observation’, Journal of Cognitive Neuroscience, vol. 26, no. 9, pp. 2087–101.

Heimann, K, Calbi, M, Uithol, S, Umiltà, MA, Guerra, M, Fingerhut, J & Gallese, V 2019, ‘Embodying the camera: an EEG study on the effect of camera movements on film spectators ́ sensorimotor cortex activation’, Plos One, vol. 14, no. 3, p. e0211026.

Hemmings, C 2005, ‘Invoking affect: cultural theory and the ontological turn’, Cultural Studies, vol. 19, no. 5, pp. 548–67.

Hidaka, S, Teramoto, W, Sugita, Y, Manaka, Y, Sakamoto, S & Suzuki, Y 2011, ‘Auditory motion information drives visual motion perception’, PLoS ONE, vol. 6, no. 3, p. 9.

Hidaka, S, Manaka, Y, Teramoto, W, Sugita, Y, Miyauchi, R, Gyoba, J, Suzuki, Y & Iwaya, Y 2009, ‘Alternation of sound location induces visual motion perception of a static object’, PLoS One, vol. 4, no. 12, p. e8188.

Hilmes, M 2005, ‘Is there a field called sound culture studies? And does it matter?’, American Quarterly, vol. 57, no. 1, pp. 249–59.

Hitchcock, A 1960, Psycho, motion picture, United States.

Hoeckner, B, Wyatt, EW, Decety, J & Nusbaum, H 2011, ‘Film music influences how viewers relate to movie characters’, Psychology of Aesthetics, Creativity, and the Arts, vol. 5, no. 2, pp. 146–53.

Hogan, PC 2003, Cognitive science, literature, and the arts: a guide for humanists, Routledge, New York; London.

---- 2008, Understanding Indian movies: culture, cognition, and cinematic imagination, University of Texas Press, Austin, TX.

317 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. ---- 2010, ‘A passion for plot: prolegomena to affective narratology’, symploke, vol. 18, no. 1-2, pp. 65–81.

---- 2011, Affective narratology: the emotional structure of stories, University of Nebraska Press, Lincoln, NE.

Holland, N 2008, ‘Spider-Man? Sure! The neuroscience of suspending disbelief’, Interdisciplinary Science Reviews, vol. 33, no. 4, pp. 312–20.

Hommel, B, Müsseler, J, Aschersleben, G & Prinz, W 2001, ‘The theory of event coding (TEC): a framework for perception and action planning’, Behavioral and Brain Sciences, vol. 24, pp. 849–937.

Hopper, D 1969, Easy Rider, motion picture, USA.

Horgan, T & Tienson, J 1996, Connectionism and the philosophy of psychology, MIT Press, Cambridge, Mass.

Horning, SS 2004, ‘Engineering the performance: recording engineers, tacit knowledge and the art of controlling sound’, Social Studies of Science, vol. 34, no. 5, pp. 703–31.

Hoshiyama, M, Atsuko, G & Kakigi, R 2001, ‘Hearing the sound of silence: a magnetoencephalographic study’, NeuroReport, vol. 12, no. 6, pp. 1097–102.

Hubbard, TL 2010, ‘Auditory imagery: empirical findings’, Psychological Bulletin, vol. 136, no. 2, pp. 302–29.

Hurley, SL 1998, Consciousness in action, Harvard University Press, Cambridge, Mass. ; London.

Huron, D 2002, A six-component theory of auditory-evoked emotion, paper presented at ICMPC7: 7th International Conference on Music Perception & Cognition, University of New South Wales, Sydney, Australia, 17–21 July 2002.

Huvenne, M 2013, Transmodal characteristics of sound and haptic aurality: creation of sound in film in the perspective of audiovisual perception, paper presented at Film-Philosophy Conference 2013: Beyond Film, ASCA (Amsterdam) and EYE Film Institute Netherlands, 10 July 2013 – 12 July 2013.

Hyvärinen, M 2008, ‘"Life as Narrative" revisited’, Partial Answers: Journal of Literature and the History of Ideas, vol. 6, no. 2, pp. 261–77.

---- 2010, ‘Revisiting the narrative turns’, Life Writing, vol. 7, no. 1, pp. 69–82.

---- 2012, ‘Prototypes, genres, and concepts: travelling with narratives’, Narrative Works: Issues, Investigations, & Interventions, vol. 2, no. 1, pp. 10–32.

Iacoboni, M 2009a, ‘Imitation, empathy, and mirror neurons’, Annual Review of Psychology, vol. 60, pp. 653– 70.

---- 2009b, ‘Neurobiology of imitation’, Current Opinion in Neurobiology, vol. 19, no. 6, pp. 661–5.

Iacoboni, M, Molnar-Szakacs, I, Gallese, V, Buccino, G, Mazziotta, JC & Rizzolatti, G 2005, ‘Grasping the intentions of others with one's own mirror neuron system’, PLoS Biology, vol. 3, no. 3, Article e79, pp. 529–35, doi 10.1371/journal.pbio.0030079, viewed 18 June 2011, .

Immediate Music 2009, Prologue to a Quest, music recording, USA.

In the Cut (cast & crew), 2003, Internet Movie Database (IMDb), viewed 27 April 2017, .

Inárritu, AG 2006, Babel, motion picture, France, USA, Mexico.

Iñárritu, AG 2015, The Revenant, motion picture.

318 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Ione, A 2006, ‘Locating the artist within views of consciousness: perception, reception, and art history’, in R Pepperell & M Punt (eds), Screen consciousness: cinema, mind and world, Rodopi, Amsterdam, pp. 27–37.

Iriki, A 2006, ‘The neural origins and implications of imitation, mirror neurons and tool use’, Current Opinion in Neurobiology, vol. 16, no. 6, pp. 660–7.

Iwerks, U 1928, Steamboat Willie, motion picture, USA.

Izard, CE 2010, ‘The many meanings/aspects of emotion: definitions, functions, activation, and regulation’, Emotion Review, vol. 2, no. 4, pp. 363–70.

Jackson, P 2005, King Kong, motion picture, New Zealand, USA, Germany.

James, W 1884, ‘What is an emotion?’, Mind, vol. 9, no. 34, pp. 188–205.

---- 1890, The principles of psychology, 2 vols., Macmillan, London.

Jarrett, M & Murch, W 2000, ‘Sound doctrine: an interview with Walter Murch’, Film Quarterly, vol. 53, no. 3, pp. 2–11.

Jay, M 1993, Downcast eyes: the denigration of vision in twentieth-century French thought, University of California Press, Berkeley.

---- 2002, ‘Cultural relativism and the visual turn’, Journal of Visual Culture, vol. 1, no. 3, pp. 267–78.

---- 2011, ‘The menace of consilience: keeping the disciplines unreconciled’, in R Foshay (ed.), Valences of interdisciplinarity: theory, practice, pedagogy, AU Press, Edmonton, pp. 31–46.

Jeannerod, M 2006, ‘Consciousness of action as an embodied consciousness’, in S Pockett, WP Banks & S Gallagher (eds), Does consciousness cause behavior?, MIT Press, Cambridge, Mass., pp. 25–38.

Jensen, CS 2014, ‘Reduced narration, intensified emotion: the film trailer’, Projections, vol. 8, no. 1, pp. 105– 25.

Johnson, L, Sutton, J & Tribble, EB (eds) 2014, Embodied cognition and Shakespeare's theatre: the early modern body-mind, Routledge, New York.

Johnson, M 1987, The body in the mind: the bodily basis of meaning, imagination, and reason, University of Chicago Press, Chicago.

---- 1997, ‘Embodied meaning and cognitive science’, in DM Levin (ed.), Language beyond postmodernism: saying and thinking in Gendlin's philosophy, Northwestern University Press, Evanston, pp. 148–75.

---- 2005, ‘The philosophical significance of image schemas’, in B Hampe & JE Grady (eds), From perception to meaning: image schemas in cognitive linguistics, Mouton de Gruyter, Berlin; New York, pp. 15–33.

---- 2007a, ‘'The stone that was cast out shall become the cornerstone': the bodily aesthetics of human meaning’, Journal of Visual Art Practice, vol. 6, no. 2, pp. 89–103.

---- 2007b, The meaning of the body: aesthetics of human understanding, University of Chicago Press, Chicago ; London.

Johnson, S 2005a, Everything bad is good for you: how today's popular culture is actually making us smarter, Penguin, London.

---- 2005b, ‘Watching TV makes you smarter’, New York Times, 24 April 2005, viewed 2 August 2016, .

319 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Johnson, W 1985, ‘The liberation of echo: a new hearing for film sound’, Film Quarterly, vol. 38, no. 4, pp. 2– 12.

---- 1989, ‘Sound and image: a further hearing’, Film Quarterly, vol. 43, no. 1, pp. 24–35.

Jolij, J & Meurs, M 2011, ‘Music alters visual perception’, PLoS One, vol. 6, no. 4, pp. 1–5.

Jomori, I, Hoshiyama, M, Uemura, J-i, Nakagawa, Y, Hoshino, A & Iwamoto, Y 2012, ‘Effects of emotional music on visual processes in inferior temporal area’, Cognitive Neuroscience, vol. 4, no. 1, pp. 21–30.

Jonze, S 2002a, Adaptation, motion picture, USA.

---- 2002b, IKEA Lamp, television commercial, USA.

---- 2013, Her, motion picture, USA.

Jordan, R 2007, ‘The work of Hildegard Westerkamp in the films of Gus Van Sant: an interview with the soundscape composer (and some added thoughts of my own)’, Offscreen, vol. 11, no. 8/9, Article 4, viewed 27 March 2013, .

Joseph, M 2015a, We Are Your Friends (trailer), online multimedia, viewed 22 June 2015

---- 2015b, We Are Your Friends, motion picture, USA.

Joy, A, Sherry Jr, JF & Deschenes, J 2009, ‘Conceptual blending in advertising’, Journal of Business Research, vol. 62, no. 1, pp. 39–49.

Juslin, PN 2013a, ‘The value of a uniquely psychological approach to musical aesthetics: Reply to the commentaries on ‘A unified theory of musical emotions’’, Physics of Life Reviews, vol. 10, no. 3, pp. 281–6.

---- 2013b, ‘From everyday emotions to aesthetic emotions: towards a unified theory of musical emotions’, Physics of Life Reviews, vol. 10, no. 3, pp. 235–66.

Juslin, PN & Västfjäll, D 2008, ‘Emotional responses to music: the need to consider underlying mechanisms’, Behavioral and Brain Sciences, vol. 31, no. 5, pp. 559–75.

Juslin, PN & Sloboda, JA 2010, ‘Introduction: aims, organization, and terminology’, in PN Juslin & JA Sloboda (eds), Handbook of music and emotion: theory, research, applications, Oxford University Press, Oxford; New York, pp. 3–12.

Juslin, PN, Harmat, L & Eerola, T 2013, ‘What makes music emotionally significant? Exploring the underlying mechanisms’, Psychology of Music, vol. 42, no. 4, pp. 599–623.

Juslin, PN, Barradas, G & Eerola, T 2015, ‘From sound to significance: exploring the mechanisms underlying emotional reactions to music’, The American Journal of Psychology, vol. 128, no. 3, pp. 281–304.

Juslin, PN, Liljeström, S, Västfjäll, D & Lundqvist, L-O 2010, ‘How does music evoke emotions? Exploring the underlying mechanisms’, in PN Juslin & JA Sloboda (eds), Handbook of music and emotion: theory, research, applications, Oxford University Press, Oxford; New York, pp. 605–42.

Juslin, PN, Liljeström, S, Laukka, P, Västfjäll, D & Lundqvist, L-O 2011, ‘Emotional reactions to music in a nationally representative sample of Swedish adults’, Musicae Scientiae, vol. 15, no. 2, pp. 174–207.

Kafaligonul, H & Stoner, GR 2010, ‘Auditory modulation of visual apparent motion with short spatial and temporal intervals’, Journal of Vision, vol. 10, no. 12, pp. 31–.

Kahn, D 1999, Noise, water, meat: a history of sound in the arts, MIT Press, Cambridge, Mass.

320 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Kalinak, K (ed.) 2015, Sound: dialogue, music, and effects, Rutgers University Press, New Brunswick, New Jersey

Kappas, A 2006, ‘Appraisals are direct, immediate, intuitive, and unwitting…and some are reflective…’, Cognition and Emotion, vol. 20, no. 7, pp. 952–75.

Kawachi, Y, Grove, PM & Sakurai, K 2014, ‘A single auditory tone alters the perception of multiple visual events’, Journal of Vision, vol. 14, no. 16.

Keane, S 2007, ‘Walter Murch and Ben Burtt: the sound designer as composer’, in G Harper, R Doughty & J Eisentraut (eds), Sound and music in film and visual media: a critical overview, Continuum, London, pp. 452–62.

Keating, P 2006, ‘Emotional curves and linear narratives’, The Velvet Light Trap, vol. 58, no. 1, pp. 4–15.

Keen, S 2011, ‘Narrative and the emotions’, Poetics Today, vol. 32, no. 1, pp. 1–53.

---- 2015, ‘Narrative emotions’, in Narrative form, 2nd edn, Palgrave Macmillan, Houndmills, UK, pp. 152–61.

Keetels, M & Vroomen, J 2011, ‘Sound affects the speed of visual processing’, Journal of Experimental Psychology: Human Perception and Performance, vol. 37, no. 3, pp. 699–708.

Kennedy, BM 2000, Deleuze and cinema: the aesthetics of sensation, Edinburgh University Press, Edinburgh.

Kenny, T 1998, ‘The search for order in sound & picture’, Mix, no. 4 (April), pp. 12–24.

Kerins, M 2006, ‘Narration in the cinema of digital sound’, The Velvet Light Trap, vol. 58, pp. 41–54.

---- 2008, ‘A Statement on Sound Studies (with apologies to Sergei Eisenstein, Vsevolod Pudovkin, and Grigori Alexandrov)’, Music, Sound, and the Moving Image, vol. 2, no. 2, pp. 115–9.

---- 2010, Beyond Dolby (stereo): cinema in the digital sound age, Indiana University Press, Bloomington.

---- 2013a, ‘Understanding the impact of surround sound in multimedia’, in S-L Tan, A Cohen, SD Lipscomb & RA Kendall (eds), Psychology of music in multimedia, Oxford University Press, New York, pp. 365–90.

---- 2013b, ‘Multichannel gaming aesthetics of interactive surround’, in J Richardson, C Gorbman & C Vernallis (eds), The Oxford Handbook of New Audiovisual Aesthetics, Oxford University Press, Oxford ; New York, pp. 585–602.

Kerr, CE 2008, ‘Dualism redux in recent reuroscience: "Theory of Mind" and "Embodied Simulation" hypotheses in light of historical debates about perception, cognition, and mind’, Review of General Psychology, vol. 12, no. 2, pp. 205–14.

Keysers, C & Gazzola, V 2009, ‘Expanding the mirror: vicarious activity for actions, emotions, and sensations’, Current Opinion in Neurobiology, vol. 19, no. 6, pp. 666–71.

Keysers, C, Kohler, E, Umiltà, MA, Nanetti, L, Fogassi, L & Gallese, V 2003, ‘Audiovisual mirror neurons and action recognition’, Experimental Brain Research, vol. 153, no. 4, pp. 628–36.

Khoo, GS & Oliver, MB 2013, ‘The therapeutic effects of narrative cinema through clarification: reexamining catharsis’, Scientific Study of Literature, vol. 3, no. 2, pp. 266–93.

Kihlstrom, JF 1984, ‘Conscious, subconscious, unconscious: a cognitive perspective’, in KS Bowers & D Meichenbaum (eds), The unconscious reconsidered, Wiley, New York, pp. 149–211.

Kihlstrom, JF, Barnhardt, TM & Tataryn, DJ 1992, ‘The psychological unconscious: found, lost, and regained’, American Psychologist, vol. 47, no. 6, pp. 788–91.

321 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Kihlstrom, JF, Mulvaney, S, Tobias, BA & Tobis, IP 2000, ‘The emotional unconscious’, in E Eich, JF Kihlstrom, GH Bower, JP Forgas & PM Niedenthal (eds), Cognition and emotion, Oxford University Press, Oxford ; New York, pp. 30–86.

Kim, R, Peters, MAK & Shams, L 2012, ‘0 + 1 > 1: how adding noninformative sound improves performance on a visual task’, Psychological Science, vol. 23, no. 1, pp. 6–12.

Kim-Cohen, S 2009, In the blink of an ear: towards a non-cochlear sonic art, Continuum, New York.

Kindt, T 2009, ‘Narratological expansionism and its discontents’, in S Heinen & R Sommer (eds), Narratology in the age of cross-disciplinary narrative research, Walter de Gruyter, Berlin; New York, pp. 35–47.

King, AJ 2006, ‘Auditory neuroscience: activating the cortex without sound’, Current Biology, vol. 16, no. 11, pp. R410–R1.

King, G 2000, Spectacular narratives: Hollywood in the age of the blockbuster, IB Tauris, London.

---- 2002, New Hollywood cinema: an introduction, New York, Columbia University Press.

---- 2003, ‘Spectacle, narrative, and the spectacular Hollywood Blockbuster’, in J Stringer (ed.), Movie blockbusters, Routledge, London, pp. 114–27.

King, N 1984, ‘The sound of silents’, Screen, vol. 25, no. 3, pp. 2–15.

Kivy, P 1990, Music alone: philosophical reflections on the purely musical experience, Cornell University Press, Ithaca, NY.

Kleinginna, PR & Kleinginna, AM 1981, ‘A categorized list of motivation definitions, with a suggestion for a consensual definition’, Motivation and Emotion, vol. 5, no. 3, pp. 263–91.

Klimov, E 1985, Come and See, motion picture, USSR.

Klingberg, T 2009, The overflowing brain: information overload and the limits of working memory, Oxford University Press, Oxford ; New York.

Knöferle, KM & Spence, C 2012, ‘Crossmodal correspondences between sounds and tastes’, Psychonomic Bulletin & Review, vol. 19, no. 6, pp. 992–1006.

Knöferle, KM, Woods, A, Käppler, F & Spence, C 2015, ‘That sounds sweet: using cross-modal correspondences to communicate gustatory attributes’, Psychology & Marketing, vol. 32, no. 1, pp. 107–20.

Knowles, A 2015, Huarache Lights, music video, UK.

Koch, C & Tsuchiya, N 2007, ‘Phenomenology without conscious access is a form of consciousness without top-down attention’, Behavioral and Brain Sciences, vol. 30, no. 5/6, pp. 509–10.

Koelsch, S & Siebel, WA 2005, ‘Towards a neural basis of music perception’, Trends in Cognitive Sciences, vol. 9, no. 12, pp. 578–84.

Koelsch, S, Fritz, T, v. Cramon, DY, Müller, K & Friederici, AD 2006, ‘Investigating emotion with music: an fMRI study’, Human Brain Mapping, vol. 27, no. 3, pp. 239–50.

Koenigs, M & Tranel, D 2007, ‘Irrational economic decision-making after ventromedial prefrontal damage: evidence from the ultimatum game’, The Journal of Neuroscience, vol. 27, no. 4, pp. 951–6.

---- 2008, ‘Prefrontal cortex damage abolishes brand-cued changes in cola preference’, Social, Cognitive, and Affective Neuroscience, vol. 3, no. 1, pp. 1–6.

322 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Koenigs, M, Young, L, Adolphs, R, Tranel, D, Cushman, F, Hauser, M & Damásio, AR 2007, ‘Damage to the prefrontal cortex increases utilitarian moral judgements’, Nature, vol. 446, no. 7138, pp. 908–11.

Kohler, E, Keysers, C, Umiltà, MA, Fogassi, L, Gallese, V & Rizzolatti, G 2002, ‘Hearing sounds, understanding actions: action representation in mirror neurons’, Science, vol. 297, no. 5582, pp. 846–8.

Kolber, D 2002, ‘Hildegard Westerkamp's Kits Beach Soundwalk: shifting perspectives in real world music’, Organised Sound, vol. 7, no. 1, pp. 41–3.

Konečni, VJ 2005, ‘The aesthetic trinity: awe, being moved, thrills’, Bulletin of Psychology and the Arts, vol. 5, no. 2, pp. 27–44.

Kosinski, J 2010, TRON: Legacy, motion picture, USA.

Kosslyn, SM 1980, Image and mind, Harvard University Press, Cambridge, MA.

---- 1981, ‘The medium and the message in mental imagery: a theory’, Psychological Review, vol. 88, no. 1, pp. 46–66.

---- 1994, Image and brain: the resolution of the imagery debate, MIT Press, Cambridge, Mass.

---- 2005, ‘Mental images and the brain’, Cognitive Neuropsychology, vol. 22, no. 3/4, pp. 333–47.

Kosslyn, SM, Ganis, G & Thompson, WL 2001, ‘Neural foundations of imagery’, Nature Reviews. Neuroscience, vol. 2, no. 9, pp. 635–42.

Kosslyn, SM, Ganis, G & Thompson, WL 2003, ‘Mental imagery: against the nihilistic hypothesis’, Trends in Cognitive Sciences, vol. 7, no. 3, pp. 109–11.

Koszarski, R 1994, An evening's entertainment: the age of the silent feature picture, 1915-1928, History of the American cinema, University of California Press, Berkeley.

Kövecses, Z 2000, Metaphor and emotion: language, culture, and body in human feeling, Cambridge University Press, Cambridge, UK.

Kracauer, S 1960, Theory of film: the redemption of physical reality, Oxford University Press, London.

Kraemer, DJM, Macrae, CN, Green, AE & Kelley, WM 2005, ‘Musical imagery: sound of silence activates auditory cortex’, Nature, vol. 434, no. 7030, p. 158.

Kreiswirth, M 1992, ‘Trusting the tale: the narrativist turn in the human sciences’, New Literary History, vol. 23, no. 3, pp. 629–57.

---- 1995, ‘Tell me a story: the narrativist turn in the human sciences’, in M Kreiswirth & TJD Carmichael (eds), Constructive criticism: the human sciences in the age of theory, University of Toronto Press, Toronto ; Buffalo, pp. 61–87.

---- 2000, ‘Merely telling stories? Narrative and knowledge in the human sciences’, Poetics Today, vol. 21, no. 2, pp. 293–318.

---- 2005, ‘Narrative turn in the humanities’, in D Herman, M Jahn & M-L Ryan (eds), Routledge encyclopedia of narrative theory, Routledge, New York, pp. 377–82.

Kromhout, M 2014, ‘“Sound Studies: a discipline?”: Sound Signatures Winter School, Amsterdam, January 2014’, Sounding Out!, Article viewed 17 December 2018, .

Krumhansl, CL 2002, ‘Music: a link between cognition and emotion’, Current Directions in Psychological Science, vol. 11, no. 2, pp. 45–50.

Krumhansl, CL & Lerdahl, F 2011, ‘Musical tension’, in F Bacci & D Melcher (eds), Art and the senses, Oxford University Press, Oxford ; New York, pp. 311–27.

Kubovy, M 1988, ‘Should we resist the seductiveness of the space:time::vision:audition analogy?’, Journal of Experimental Psychology: Human Perception and Performance, vol. 14, no. 2, pp. 318–20.

Kubrick, S 1968, 2001: a space odyssey, motion picture, UK, USA.

Kuipers, B 2008, ‘Drinking from the firehose of experience’, Artificial Intelligence in Medicine, vol. 44, no. 2, pp. 155–70.

Kulezic-Wilson, D 2008, ‘Sound design is the new score’, Music, Sound & the Moving Image, vol. 2, no. 2, pp. 127–31.

---- 2015, The musicality of narrative film, Palgrave Macmillan, Houndmills, Basingstoke, Hampshire ; New York.

LaBelle, B 2006, Background noise: perspectives on sound art, Continuum, New York.

---- 2010, Acoustic territories: sound culture and everyday life, Continuum, New York.

Lahav, A, Saltzman, E & Schlaug, G 2007, ‘Action representation of sound: audiomotor recognition network while listening to newly acquired actions’, Journal of Neuroscience, vol. 27, no. 2, pp. 308–14.

Laine, T 2006, ‘Cinema as second skin’, New Review of Film and Television Studies, vol. 4, no. 2, pp. 93–106.

---- 2007, ‘"It's the sense of touch": skin in the making of cinematic consciousness’, Discourse: Journal for Theoretical Studies in Media & Culture, vol. 29, no. 1, pp. 35–48.

---- 2011, Feeling cinema: emotional dynamics in Film Studies, Continuum, New York.

Laine, T & Strauven, W 2009, ‘The synaesthetic turn’, New Review of Film and Television Studies, vol. 7, no. 3, pp. 249–55.

Lakoff, G 1987, Women, fire, and dangerous things: what categories reveal about the mind, University of Chicago Press, Chicago.

Lakoff, G & Johnson, M 1980a, Metaphors we live by, University of Chicago Press, Chicago.

---- 1980b, ‘Conceptual metaphor in everyday language’, The Journal of Philosophy, vol. 77, no. 8, pp. 453–86.

---- 1999, Philosophy in the flesh: the embodied mind and its challenge to Western thought, Basic Books, New York.

Laland, KN, Kumm, J & Feldman, MW 1995, ‘Gene-culture coevolutionary theory: a test case’, Current Anthropology, vol. 36, no. 1, pp. 131–56.

Lamarque, P 2004, ‘On not expecting too much from narrative’, Mind & Language, vol. 19, no. 4, pp. 393–408.

Langdale, A (ed.) 2002, Hugo Münsterberg on film: The photoplay: a psychological study, and other writings, Routledge, New York.

Langkjær, B & Have, I 2012, ‘Conceptualising sound’, SoundEffects – an Interdisciplinary Journal of Sound and Sound Experience, vol. 2, no. 2, Article 1, pp. 2–3, viewed 3 November 2014, .

Lapate, RC, Rokers, B, Li, T & Davidson, RJ 2014, ‘Nonconscious emotional activation colors first impressions: a regulatory role for conscious awareness’, Psychological Science, vol. 25, no. 2, pp. 349–57.

324 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Lastra, J 2000, Sound technology and the American cinema: perception, representation, modernity, Columbia University Press, New York.

Lau, H & Rosenthal, D 2011, ‘Empirical support for higher-order theories of conscious awareness’, Trends in Cognitive Sciences, vol. 15, no. 8, pp. 365–73.

Lavik, E 2008, ‘The battle for the blockbuster: discourses of spectacle and excess’, New Review of Film and Television Studies, vol. 6, no. 2, pp. 169–87.

Lawrence, A 1991, Echo and Narcissus: women's voices in classical Hollywood cinema, University of California Press, Berkeley.

Lawrence, R 2001, Lantana, motion picture, Australia, Germany.

Lazarus, RS 1982, ‘Thoughts on the relations between emotion and cognition’, American Psychologist, vol. 37, no. 9, pp. 1019–24.

---- 1984, ‘On the primacy of cognition’, American Psychologist, vol. 39, no. 2, pp. 124–9.

---- 1991, Emotion and adaptation, Oxford University Press, New York.

---- 1999, ‘The cognition-emotion debate: a bit of history’, in T Dalgleish & MJ Power (eds), Handbook of cognition and emotion, John Wiley & Sons Ltd, New York, NY, pp. 3–19.

Le Bel, RM, Pineda, JA & Sharma, A 2009, ‘Motor-auditory-visual integration: the role of the human mirror neuron system in communication and communication disorders’, Journal of Communication Disorders, vol. 42, no. 4, pp. 299–304.

Le Prince, LAA 1888a, Traffic Crossing Leeds Bridge, motion picture, UK.

---- 1888b, Roundhay Garden Scene, motion picture, UK.

Leder, D 1990, The absent body, University of Chicago Press, Chicago.

LeDoux, JE 1986, ‘Sensory systems and emotion: a model of affective processing’, Integrative Psychiatry, vol. 4, no. 4, pp. 237–43.

---- 1989, ‘Cognitive-emotional interactions in the brain’, Cognition and Emotion, vol. 3, no. 4, pp. 267–89.

---- 1991, ‘Emotion and the limbic system concept’, Concepts in Neuroscience, vol. 2, pp. 169–99.

---- 1994, ‘Emotional processing, but not emotions, can occur unconsciously’, The nature of emotion: Fundamental questions, pp. 291–2.

---- 1995, ‘Emotion: clues from the brain’, Annual Review of Psychology, vol. 46, pp. 209–35.

---- 1996, The emotional brain: the mysterious underpinnings of emotional life, Simon & Schuster, New York.

---- 2000, ‘Emotion circuits in the brain’, Annual Review of Neuroscience, vol. 23, no. 1, pp. 155–84.

---- 2007, ‘Unconscious and conscious contributions to the emotional and cognitive aspects of emotions: a comment on Scherer's view of what an emotion is’, Social Science Information, vol. 46, no. 3, pp. 395– 405.

LeDoux, JE & Phelps, EA 1993, ‘Emotional networks in the brain’, in M Lewis & JM Haviland-Jones (eds), Handbook of emotions, 2nd edn, Guilford Press, New York, pp. 109–18.

Lee, KM 2004a, ‘Why presence occurs: evolutionary psychology, media equation, and presence’, Presence: Teleoperators & Virtual Environments, vol. 13, no. 4, pp. 494–505.

Léger, F & Murphy, D 1924, Ballet Mécanique, motion picture, Austria.

Leo, F, Romei, V, Freeman, E, Ladavas, E & Driver, J 2011, ‘Looming sounds enhance orientation sensitivity for visual stimuli on the same side as such sounds’, Experimental Brain Research, vol. 213, no. 2, pp. 193–201.

Lerdahl, F & Krumhansl, CL 2007, ‘Modeling tonal tension’, Music Perception, vol. 24, no. 4, pp. 329–66.

Lessing, GE 1874 [1766], Laocoön: an essay upon the limits of poetry and painting, Roberts Brothers, Boston.

Levin, DM 1993a, ‘Introduction’, in DM Levin (ed.), Modernity and the hegemony of vision, University of California Press, Berkeley, pp. 1–29.

---- (ed.) 1993b, Modernity and the hegemony of vision, University of California Press, Berkeley.

Levine, J 1983, ‘Materialism and qualia: the explanatory gap’, Pacific Philosophical Quarterly, vol. 64, pp. 354– 61.

Levinson, SC & Majid, A 2014, ‘Differential ineffability and the senses’, Mind & Language, vol. 29, no. 4, pp. 407–27.

Levitin, DJ 2006, This is your brain on music: the science of a human obsession, Dutton, New York.

Lewis, JW, Wightman, FL, Brefczynski, JA, Phinney, RE, Binder, JR & DeYoe, EA 2004, ‘Human brain regions involved in recognizing environmental sounds’, Cerebral Cortex, vol. 14, no. 9, pp. 1008–21.

Lewis, M 2015, ‘Ventriloquial acts: critical reflections on the art of Foley’, The New Soundtrack, vol. 5, no. 2, pp. 103–20.

Lewis, S 2014, ‘What is spectacle?’, Journal of Popular Film and Television, vol. 42, no. 4, pp. 214–21.

Lewkowicz, DJ 1988a, ‘Sensory dominance in infants: I. Six-month-old infants' response to auditory-visual compounds’, Developmental Psychology, vol. 24, no. 2, pp. 155–71.

---- 1988b, ‘Sensory dominance in infants: II. Ten-month-old infants' response to auditory-visual compounds’, Developmental Psychology, vol. 24, no. 2, pp. 172–82.

Leydon, R 2012, ‘Clean as a whistle: timbral trajectories and the modern musical sublime’, Music Theory Online, vol. 18, no. 2, pp. 1–17.

Leys, R 2011, ‘The turn to affect: a critique’, Critical Inquiry, vol. 37, no. 3, pp. 434–72.

Li, S-C 2003, ‘Biocultural orchestration of developmental plasticity across levels: the interplay of biology and culture in shaping the mind and behavior across the lifespan’, Psychological Bulletin, vol. 129, no. 2, pp. 171–94.

Lieberman, MD 2010, ‘Social cognitive neuroscience’, in ST Fiske, DT Gilbert & G Lindzey (eds), Handbook of social psychology, John Wiley & Sons, Hoboken, NJ, pp. 143–93.

Lindquist, KA 2013, ‘Emotions emerge from more basic psychological ingredients: a modern psychological constructionist model’, Emotion Review, vol. 5, no. 4, pp. 356–68.

Lindsay, V 1915, Art of the moving picture, The Macmillan Company, New York.

Lipscomb, SD 1999, ‘Cross-modal integration: synchronization of auditory and visual components in simple and complex media’, The Journal of the Acoustical Society of America, vol. 105, no. 2, p. 1274.

326 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. ---- 2013, ‘Cross-modal alignment of accent structures in multimedia’, in S-L Tan, A Cohen, SD Lipscomb & RA Kendall (eds), The psychology of music in multimedia, Oxford University Press, Oxford, UK, pp. 192–215.

Lisberger, S 1982, TRON, motion picture, USA.

LoBrutto, V 1994, Sound-on-film: interviews with creators of film sound, Praeger, Westport, CT.

Logeswaran, N & Bhattacharya, J 2009, ‘Crossmodal transfer of emotion by music’, Neuroscience Letters, vol. 455, no. 2, pp. 129–33.

Lombard, M & Ditton, T 1997, ‘At the heart of it all: the concept of presence’, Journal of Computer-Mediated Communication, vol. 3, no. 2, Article 1, doi 10.1111/j.1083-6101.1997.tb00072.x, viewed 12 May 2019, .

Lords of Acid 1994, The Crablouse, music recording, USA.

Loughney, P 1999, ‘Domitor witnesses the first complete public presentation of the 'Dickson experimental sound film' in the 20th century’, Film History, vol. 11, no. 4, pp. 400–3.

Lovell, A & Sergi, G 2009, ‘What audiences go for: elite and mass taste’, in Cinema entertainment: essays on audiences, films and film-makers, Open University Press, Maidenhead, England ; New York, pp. 4–21.

Lubart, TI & Getz, I 1997, ‘Emotion, metaphor, and the creative process’, Creativity Research Journal, vol. 10, no. 4, pp. 285–301.

Lucas, G 1971, THX 1138, motion picture, USA.

---- 1973, American Graffiti, motion picture, USA.

---- 1977, Star Wars, motion picture, USA.

Lumière, A & Lumière, L 1895, L'Arrivée d'un train en gare de La Ciotat, motion picture, France.

Lumière, L 1895, La Sortie de l'Usine Lumière à Lyon, motion picture, France.

Lumsden, CJ & Wilson, EO 1981, Genes, mind, and culture: the coevolutionary process, Harvard University Press, Cambridge, Mass.

Lutz, A & Thompson, E 2003, ‘Neurophenomenology: integrating subjective experience and brain dynamics in the neuroscience of consciousness’, Journal of Consciousness Studies, vol. 10, no. 9-10, pp. 31–52.

Lutz, C 1988, Unnatural emotions: everyday sentiments on a Micronesian atoll & their challenge to western theory, University of Chicago Press, Chicago.

Lycan, WG 1996, Consciousness and experience, MIT Press, Cambridge, Mass.

---- 2001, ‘A simple argument for a higher-order representation theory of consciousness’, Analysis, vol. 61, no. 1, pp. 3–4.

Lynch, D 1977, Eraserhead, motion picture, USA.

---- 1997, Lost Highway, motion picture, France, USA.

---- 2001, Mulholland Drive, motion picture, France, USA.

Ma, W & Thompson, WF 2015, ‘Human emotions track changes in the acoustic environment’, Proceedings of the National Academy of Sciences, vol. 112, no. 47, pp. 14563–8.

327 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Macallan, H & Plain, A 2010, ‘Filmic voices’, in N Neumark, R Gibson & TV Leeuwen (eds), Voice: vocal aesthetics in digital arts and media, MIT Press, Cambridge, Mass., pp. 243–66.

Mad Men, 2007–2015, television drama, Lionsgate Television, USA.

Maeda, F, Kanai, R & Shimojo, S 2004, ‘Changing pitch induced visual motion illusion’, Current Biology, vol. 14, no. 23, pp. R990–R1.

Magliano, JP & Zacks, JM 2011, ‘The impact of continuity editing in narrative film on event segmentation’, Cognitive Science, vol. 35, no. 8, pp. 1489–517.

Maltby, R 1998, ‘‘Nobody knows everything’: post-classical historiographies and consolidated entertainment’, in S Neale & M Smith (eds), Contemporary Hollywood cinema, Routledge, London; New York, pp. 21–44.

Mankiewicz, JL 1949, A Letter to Three Wives, motion picture, USA.

Manolas, C & Pauletto, S 2009, ‘Enlarging the diegetic space: uses of the multi-channel soundtrack in cinematic narrative’, The Soundtrack, vol. 2, no. 1, pp. 39–55.

Mar, RA & Oatley, K 2008, ‘The function of fiction is the abstraction and simulation of social experience’, Perspectives on Psychological Science, vol. 3, no. 3, pp. 173–92.

Maras, S & Sutton, D 2000, ‘Medium specificity re-visited’, Convergence, vol. 6, no. 2, pp. 98–113.

Marks, LE 1978, The unity of the senses: interrelations among the modalities, Academic Press, New York.

Marks, LU 2000, The skin of the film: intercultural cinema, embodiment, and the senses, Duke University Press.

---- 2002, Touch: sensuous theory and multisensory media, University of Minnesota Press, Minneapolis.

Martin, A 2000, ‘Lang’s sound’, in P Brophy (ed.), Cinesonic: cinema and the sound of music, Australian Film Television & Radio School; Allen & Unwin, St Leonards, pp. 155–72.

---- 2002, ‘“It has come to my ears”: Fritz Lang’s sound design’, Screening the Past, Article 1, viewed 8 July 2017, .

Martiny, KM 2011, ‘Book review of Lawrence Shapiro’s Embodied Cognition’, Phenomenology and the Cognitive Sciences, vol. 10, no. 2, pp. 297–305.

Massumi, B 1995, ‘The autonomy of affect’, Cultural Critique, vol. 31, pp. 83–109.

---- 2002a, Parables for the virtual: movement, affect, sensation, Duke University Press, Durham, NC.

---- (ed.) 2002b, A shock to thought: expressions after Deleuze and Guattari, Routledge, London.

Mastoropoulou, G, Debattista, K, Chalmers, A & Troscianko, T 2005, Auditory bias of visual attention for perceptually-guided selective rendering of animations, paper presented at GRAPHITE '05: Third International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, Dunedin, New Zealand, 29 November–2 December 2005

Mastoropoulou, G, Debattista, K, Chalmers, A & Troscianko, T 2005, ‘The influence of sound effects on the perceived smoothness of rendered animations’, paper presented to APGV’05: SIGGRAPH Second Symposium on Applied Perception in Graphics and Visualization, A Coruña, Spain, 26–28 August 2005.

Mateas, M & Stern, A 2002, ‘Towards integrating plot and character for interactive drama’, in K Dautenhahn, A Bond, L Cañamero & B Edmonds (eds), Socially Intelligent Agents, Kluwer Academic Publishers, Boston; Dordrecht; London, pp. 221–8.

---- 2005a, InteractiveStory.net, viewed 3 May 2017, .

---- 2005b, Façade, computergame, Procedural Arts, USA.

Mattern, S & Salmon, B 2008, ‘Sound Studies: framing noise’, Music, Sound, and the Moving Image, vol. 2, no. 2, pp. 139–44.

McConachie, BA 2001, ‘Doing things with image schemas: the cognitive turn in theatre studies and the problem of experience for historians’, Theatre Journal, vol. 53, no. 4, pp. 569–94.

---- 2008, Engaging audiences: a cognition approach to spectating in the theatre, Palgrave MacMillan, New York.

McConachie, BA & Hart, FE (eds) 2006, Performance and cognition: theatre studies and the cognitive turn, Routledge, New York; London.

McCrae, RR 2007, ‘Aesthetic chills as a universal marker of openness to experience’, Motivation & Emotion, vol. 31, no. 1, pp. 5–11.

McDonald, JJ, Teder-Salejarvi, WA & Hillyard, SA 2000, ‘Involuntary orienting to sound improves visual perception’, Nature, vol. 407, no. 6806, pp. 906–8.

McGill, AC 2008, ‘The contemporary Hollywood film soundtrack: professional practices and sonic styles since the 1970s’, Doctor of Philosophy, University of Exeter.

McGuire, A Portishead, Amazon.co.uk, viewed 6 September 2016, .

McGurk, H & MacDonald, J 1976, ‘Hearing lips and seeing voices’, Nature, vol. 264, no. 5588, pp. 746–8.

McKee, R 1998, Story: substance, structure, style, and the principles of screenwriting, Methuen, London.

McKernan, L 2007, ‘'Only the screen was silent ...': memories of children's cinema-going in London before the First World War’, Film Studies, no. 10, pp. 1–20.

McQueen, A 2013, ‘‘Bring the noise!’ Sonic intensified continuity in the films of Edgar Wright’, Music, Sound, and the Moving Image, vol. 7, no. 2, pp. 141–66.

Mellmann, K 2011, ‘Evolutionary psychology as a heuristic in literary studies’, in N Saul & SJ James (eds), The evolution of literature: legacies of Darwin in European cultures, Rodopi, Amsterdam, The Netherlands, pp. 299–317.

---- 2012, ‘Is storytelling a biological adaptation. Preliminary thoughts on how to pose that question’, in C Gansel & D Vanderbeke (eds), Telling stories: literature and evolution / Geschichten erzählen: literatur und evolution, De Gruyter, Berlin, pp. 30–49.

Menary, R 2010, ‘Introduction to the special issue on 4E cognition’, Phenomenology and the Cognitive Sciences, vol. 9, no. 4, pp. 459–63.

Menon, V & Levitin, DJ 2005, ‘The rewards of music listening: response and physiological connectivity of the mesolimbic system’, NeuroImage, vol. 28, no. 1, pp. 175–84.

Mera, M, Sadoff, R & Winters, B (eds) 2017, The Routledge companion to screen music and sound, Routledge, New York, NY ; London.

Metz, C 1975, ‘The imaginary signifier’, Screen, vol. 16, no. 2, pp. 14–76.

---- 1982, The imaginary signifier: psychoanalysis and the cinema, Indiana University Press, Bloomington.

329 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Meyer, GF & Wuerger, SM 2001, ‘Cross-modal integration of auditory and visual motion signals’, NeuroReport, vol. 12, no. 11, pp. 2557–60.

Meyer, K, Kaplan, JT, Essex, R, Webber, C, Damásio, H & Damásio, AR 2010, ‘Predicting visual stimuli on the basis of activity in auditory cortices’, Nature Neuroscience, vol. 13, no. 6, pp. 667–8.

Meyer, M, Baumann, S & Jancke, L 2006, ‘Electrical brain imaging reveals spatio-temporal dynamics of timbre perception in humans’, NeuroImage, vol. 32, no. 4, pp. 1510–23.

Miall, DS 1988, ‘Affect and narrative: a model of response to stories’, Poetics, vol. 17, no. 3, pp. 259–72.

---- 2011, ‘Emotions and the structuring of narrative responses’, Poetics Today, vol. 32, no. 2, pp. 323–48.

Mildorf, J & Kinzel, T 2016a, ‘Audionarratology: prolegomena to a research paradigm exploring sound and narrative’, in J Mildorf & T Kinzel (eds), Audionarratology: interfaces of sound and narrative, De Gruyter, Berlin; Boston, pp. 1–26.

---- (eds) 2016b, Audionarratology: interfaces of sound and narrative, De Gruyter, Berlin; Boston.

Miller, DW & Marks, LJ 1992, ‘Mental imagery and sound effects in radio commercials’, Journal of Advertising, vol. 21, no. 4, pp. 83–93.

Miller, G 2015, Mad Max: Fury Road, motion picture, Australia, USA.

Miller, P 1982, ‘Music and the silent film’, Perspectives of New Music, vol. 21, no. 1/2, pp. 582–4.

Minghella, A 1996, The English Patient, motion picture, USA, UK.

Mitchell, WJT 2005, ‘There are no visual media’, Journal of Visual Culture, vol. 4, no. 2, pp. 257–66.

Mithen, SJ 2006, The singing neanderthals: the origin of music, language, mind and body, Phoenix, London.

Mollaghan, A 2015, The visual music film, Palgrave Macmillan, Basingstoke, UK.

Molnar-Szakacs, I & Overy, K 2006, ‘Music and mirror neurons: from motion to 'e'motion’, Social Cognitive and Affective Neuroscience, vol. 1, no. 3, pp. 235–41.

Montag, C, Gallinat, J & Heinz, A 2008, ‘Theodor Lipps and the concept of empathy: 1851–1914’, American Journal of Psychiatry, vol. 165, no. 10, p. 1261.

Moore, S 1999, In the cut, Plume, New York.

Morris, E 1999, Mr. Death: The Rise and Fall of Fred A. Leuchter, Jr., motion picture, USA.

---- 2003, The Fog of War: Eleven Lessons from the Life of Robert S. McNamara, motion picture, USA.

Mott, RL 1993, Radio sound effects: who did it, and how, in the era of live broadcasting, McFarland, Jefferson, N.C.

Mulvey, L 1975, ‘Visual pleasure and narrative cinema’, Screen, vol. 16, no. 3, pp. 6–18.

Münsterberg, H 1916, The film: a psychological study; the silent photoplay in 1916, Dover Publications, New York.

---- 2002 [1915], ‘Why we go to the movies’, in A Langdale (ed.), Hugo Münsterberg on film: The photoplay: a psychological study, and other writings, Routledge, New York, pp. 171–90.

Murch, W 1992, In the blink of an eye: a perspective on film editing, Australian Film, Television and Radio School, Sydney, Australia.

---- 2005, Dense clarity – clear density, The Transom Review, viewed 8 July 2017, .

---- 2017, Walter Murch on the difference between sound editing and mixing, online multimedia, viewed 10 February 2017

Murnau, FW 1927, Sunrise: A Song of Two Humans, motion picture, USA.

Murphie, A 2010, ‘Deleuze, Guattari and neuroscience’, in P Gaffney (ed.), The force of the virtual: Deleuze, science, and philosophy, University of Minnesota Press, Minneapolis, pp. 330–67.

Murphy, ST & Zajonc, RB 1993, ‘Affect, cognition, and awareness: affective priming with optimal and suboptimal stimulus exposures’, Journal of Personality and Social Psychology, vol. 64, no. 5, pp. 723– 39.

Murray, MM & Wallace, MT (eds) 2011, The neural bases of multisensory processes, CRC Press, Boca Raton, FL.

Musser, C 1991, Before the nickelodeon: Edwin S. Porter and the Edison Manufacturing Company, UCLA Film and Television Archive studies in history, criticism, and theory, University of California Press, Berkeley.

---- 1994a, ‘Rethinking early cinema: cinema of attractions and narrativity’, The Yale Journal of Criticism, vol. 7, no. 2, pp. 203–32.

---- 1994b, The emergence of cinema: the American screen to 1907, vol. 1, History of the American cinema, University of California Press, Berkeley.

Mustovic, H, Scheffler, K, Di Salle, F, Esposito, F, Neuhoff, JG, Hennig, J & Seifritz, E 2003, ‘Temporal integration of sequential auditory events: silent period in sound pattern activates human planum temporale’, NeuroImage, vol. 20, no. 1, pp. 429–34.

Nagel, F, Kopiez, R, Grewe, O & Altenmüller, E 2008, ‘Psychoacoustical correlates of musically induced chills’, Musicae Scientiae, vol. 12, no. 1, pp. 101–13.

Nagel, T 1974, ‘What is it like to be a bat?’, The Philosophical Review, vol. 83, no. 4, pp. 435–50.

Nalbantian, S 2008, ‘Neuroaesthetics: neuroscientific theory and illustration from the arts’, Interdisciplinary Science Reviews, vol. 33, no. 4, pp. 357–68.

Nannicelli, T & Taberham, P (eds) 2014, Cognitive media theory, Routledge, New York; London,

Napolitano, AC 2006, ‘Modality dominance in young children: underlying mechanisms and broader implications’, unpublished doctoral disertation, The Ohio State University.

Napolitano, AC & Sloutsky, VM 2003, Flexible attention and modality preference in young children, paper presented at XXV Annual Conference of the Cognitive Science Society, Boston, Massachusetts, USA, 31 July - 2 August, 2003.

---- 2004, ‘Is a picture worth a thousand words? The flexible nature of modality dominance in young children’, Child Development, vol. 75, no. 6, pp. 1850–70.

Nava, E & Pavani, F 2013, ‘Changes in sensory dominance during childhood: converging evidence from the Colavita effect and the sound-induced flash illusion’, Child Development, vol. 84, no. 2, pp. 604–16.

Ndalianis, A 2000, ‘The frenzy of the visible: spectacle and motion in the era of the digital’, Senses of Cinema, no. 3, Article 3, viewed 18 May 2017, .

Neuendorf, KA & Lieberman, EA 2010, ‘Film: the original immersive medium’, in CC Bracken & P Skalski (eds), Immersed in media: telepresence in everyday life, Routledge, New York, pp. 9–38.

Neuhoff, JG 2001, ‘An adaptive bias in the perception of looming auditory motion’, Ecological Psychology, vol. 13, no. 2, pp. 87–110.

---- 2004, ‘Auditory motion and localization’, in JG Neuhoff (ed.), Ecological psychoacoustics, Elsevier Academic Press, Boston, pp. 87–111.

Neumann, R & Strack, F 2000, ‘"Mood contagion": the automatic transfer of mood between persons’, Journal of Personality and Social Psychology, vol. 79, no. 2, pp. 211–23.

Neumark, N, Gibson, R & Leeuwen, TV (eds) 2010, Voice: vocal aesthetics in digital arts and media, MIT Press, Cambridge, Mass.

Neumeyer, D (ed.) 2014, The Oxford handbook of film music studies, Oxford University Press, Oxford ; New York.

Niedenthal, PM 1990, ‘Implicit perception of affective information’, Journal of Experimental Social Psychology, vol. 26, no. 6, pp. 505–27.

---- 2007, ‘Embodying emotion’, Science, vol. 316, no. 5827, pp. 1002–5.

---- 2010, ‘What Bob Zajonc taught us: an introduction to the teacher and the teachings’, Emotion Review, vol. 2, no. 4, pp. 315–9.

Niedenthal, PM & Maringer, M 2009, ‘Embodied emotion considered’, Emotion Review, vol. 1, no. 2, pp. 122– 8.

Niedenthal, PM, Augustinova, M & Rychlowska, M 2010, ‘Body and mind: Zajonc's (re)introduction of the motor system to emotion and cognition’, Emotion Review, vol. 2, no. 4, pp. 340–7.

Nisbett, RE & Wilson, TD 1977, ‘Telling more than we can know: verbal reports on mental processes’, Psychological Review, vol. 84, no. 3, pp. 231–59.

Noë, A 2000, ‘Experience and experiment in art’, Journal of Consciousness Studies, vol. 7, no. 8/9, pp. 123–36.

---- 2004, Action in perception, MIT Press, Cambridge, Mass.

---- 2009, Out of our heads: why you are not your brain, and other lessons from the biology of consciousness, Hill & Wang, New York.

Noë, A & Thompson, E 2002, Vision and mind: selected readings in the philosophy of perception, MIT Press, Cambridge, Mass.

Nolan, C 2014, Interstellar, motion picture, USA, UK.

Norman, K 2004, ‘The same trail twice: Talking Rain with Hildegard Westerkamp’, in Sounding art: eight literary excursions on electronic music, Ashgate, Aldershot, England ; Burlington, VT, pp. 75–99.

North, AC 2012, ‘The effect of background music on the taste of wine’, British Journal of Psychology, vol. 103, no. 3, pp. 293–301.

Noulhiane, M, Mella, N, Samson, S, Ragot, R & Pouthas, V 2007, ‘How emotional auditory stimuli modulate time perception’, Emotion, vol. 7, no. 4, pp. 697–704.

Nyre, L 2010, ‘Sound Studies is still tuning in’, New Media & Society, vol. 12, no. 8, pp. 1388–93.

332 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. O' Brien, C 2005, Cinema's conversion to sound technology and film style in France and the U.S., Indiana University Press, Bloomington.

O'Callaghan, C 2007, Sounds: a philosophical theory, Oxford University Press, Oxford.

O'Falt, C 2019, ‘‘Roma’: Alfonso Cuarón proved Dolby Atmos is the perfect tool for intimate films, not blockbusters’, Indiewire, Article viewed 11 February 2019, .

O'Regan, JK & Noë, A 2001, ‘A sensorimotor account of vision and visual consciousness’, Behavioral and Brain Sciences, vol. 24, no. 5, pp. 939–1031.

Oatley, K 2009, ‘Communications to self and others: emotional experience and its skills’, Emotion Review, vol. 1, no. 3, pp. 206–13.

Oatley, K & Johnson-Laird, PN 1987, ‘Towards a cognitive theory of emotions’, Cognition and Emotion, vol. 1, no. 1, pp. 29–50.

Oberman, LM & Ramachandran, VS 2007, ‘The simulating social mind: the role of the mirror neuron system and simulation in the social and communicative deficits of autism spectrum disorders’, Psychological Bulletin, vol. 133, no. 2, pp. 310–27.

Ocampo, B & Kritikos, A 2011, ‘Interpreting actions: the goal behind mirror neuron function’, Brain Research Reviews, vol. 67, no. 1-2, pp. 260–7.

Ochsner, KN, Ray, RR, Hughes, B, McRae, K, Cooper, JC, Weber, J, Gabrieli, JDE & Gross, JJ 2009, ‘Bottom- up and top-down processes in emotion generation: common and distinct neural mechanisms’, Psychological Science, vol. 20, no. 11, pp. 1322–31.

Odgaard, EC, Arieh, Y & Marks, LE 2003, ‘Cross-modal enhancement of perceived brightness: sensory interaction versus response bias’, Perception & Psychophysics, vol. 65, no. 1, pp. 123–32.

---- 2004, ‘Brighter noise: sensory enhancement of perceived loudness by concurrent visual stimulation’, Cognitive, Affective & Behavioral Neuroscience, vol. 4, no. 2, pp. 127–32.

Ohler, P & Nieding, G 2006a, ‘Why play? An evolutionary perspective’, in P Vorderer & J Bryant (eds), Playing video games: motives, responses, and consequences, Lawrence Erlbaum Associates, Mahwah, N.J., pp. 101–13.

---- 2006b, ‘An evolutionary perspective on entertainment’, in J Bryant & P Vorderer (eds), Psychology of entertainment, Lawrence Erlbaum Associates, Mahwah, N.J., pp. 423–33.

Öhman, A 2002, ‘Automaticity and the amygdala: nonconscious responses to emotional faces’, Current Directions in Psychological Science, vol. 11, no. 2, pp. 62–6.

Öhman, A & Wiens, S 2003, ‘On the automaticity of autonomic responses in emotion: an evolutionary perspective’, in RJ Davidson, KR Scherer & HH Goldsmith (eds), Handbook of affective sciences, Oxford University Press, New York, pp. 256–75.

Öhman, A, Flykt, A & Lundqvist, D 2000, ‘Unconscious emotion: evolutionary perspectives, psychophysiological data and neuropsychological mechanisms’, in RD Lane, L Nadel & G Ahern (eds), Cognitive neuroscience of emotion, Oxford University Press, New York ; Oxford, pp. 296–327.

Öhman, A, Flykt, A & Esteves, F 2001, ‘Emotion drives attention: detecting the snake in the grass’, Journal of Experimental Psychology: General, vol. 130, no. 3, pp. 466–78.

Öhman, A, Carlsson, K, Lundqvist, D & Ingvar, M 2007, ‘On the unconscious subcortical origin of human fear’, Physiology & Behavior, vol. 92, no. 1-2, pp. 180–5.

333 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Okon-Singer, H, Hendler, T, Pessoa, L & Shackman, AJ 2015, ‘The neurobiology of emotion–cognition interactions: fundamental questions and strategies for future research’, Frontiers in Human Neuroscience, vol. 9, Article 58, doi 10.3389/fnhum.2015.00058, viewed 17 February 2015, .

Olivers, CNL & Van der Burg, E 2008, ‘Bleeping you out of the blink: sound saves vision from oblivion’, Brain Research, vol. 1242, pp. 191–9.

Ong, WJ 1991, Orality and literacy: the technologizing of the word, Routledge, London ; New York.

Onians, J 2007, Neuroarthistory: from Aristotle and Pliny to Baxandall and Zeki, Yale University Press, New Haven ; London.

---- 2011, ‘Neuroarthistory: reuniting ancient traditions in a new scientific approach to the understanding of art’, in M Burguete & L Lam (eds), Arts: a science matter, World Scientific Publishing Company, Singapore; Hackensack, N.J., pp. 78–98.

Ortony, A, Clore, GL & Collins, A 1988, The cognitive structure of emotions, Cambridge University Press, New York.

Oudart, J-P 1990, ‘The reality effect’, in N Browne (ed.), Cahiers du Cinema, London, Routledge, vol. 3, pp. 189–202.

Overton, WF 2002, ‘Understanding, explanation, and reductionism: finding a cure for Cartesian anxiety’, in T Brown & L Smith (eds), Reductionism and the development of knowledge, Lawrence Erlbaum Associates, Mahwah, NJ, pp. 29–51.

Pallasmaa, J 2005, The eyes of the skin: architecture and the senses [Juhani Pallasmaa], 2nd edn, John Wiley & Sons, Chichester, England.

Panksepp, J 1995, ‘The emotional sources of "chills" induced by music’, Music Perception, vol. 13, no. 2, pp. 171–207.

---- 1998a, ‘The periconscious substrates of consciousness: affective states and the evolutionary origins of the SELF’, Journal of Consciousness Studies, vol. 5, no. 6, pp. 566–82.

---- 1998b, Affective neuroscience: the foundations of human and animal emotions, Oxford University Press, New York ; Oxford.

---- 2001, ‘The neuro-evolutionary cusp between emotions and cognitions: implications for understanding consciousness and the emergence of a unified mind science’, Consciousness & Emotion, vol. 7, no. 2, pp. 141–63.

---- 2005, ‘Affective consciousness: core emotional feelings in animals and humans’, Consciousness and Cognition, vol. 14, no. 1, pp. 30–80.

Panksepp, J & Bernatzky, G 2002, ‘Emotional sounds and the brain: the neuro-affective foundations of musical appreciation’, Behavioural Processes, vol. 60, no. 2, pp. 133–55.

Papoulias, C & Callard, F 2010, ‘Biology's gift: interrogating the turn to affect’, Body & Society, vol. 16, no. 1, pp. 29–56.

Parise, CV & Spence, C 2009, ‘‘When birds of a feather flock together’: synesthetic correspondences modulate audiovisual integration in non-synesthetes’, PLoS ONE, vol. 4, no. 5, pp. 1–7.

Parke, R, Chew, E & Kyriakakis, C 2007, ‘Quantitative and visual analysis of the impact of music on perceived emotion of film’, Computers in Entertainment, vol. 5, no. 3, pp. 1–60.

Parvizi, J & Damásio, AR 2001, ‘Consciousness and the brainstem’, Cognition, vol. 79, no. 1–2, pp. 135–60.

334 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Pasquariello, N 1996, Sounds of movies: interviews with the creators of feature sound tracks, Port Bridge Books, San Francisco, CA.

Pauletto, S 2012, ‘The sound design of cinematic voices’, The New Soundtrack, vol. 2, no. 2, pp. 127–42.

Penn, A 1967, Bonnie and Clyde, motion picture, USA.

Perron, B 2001, ‘The first transi-sounds of parallel editing’, in R Abel & R Altman (eds), The sounds of early cinema, Indiana University Press, Bloomington, IN, pp. 79–86.

Pessoa, L 2013, The cognitive-emotional brain: from interactions to integration, MIT Press, Cambridge, MA.

Pessoa, L, Thompson, E & Noë, A 1998, ‘Finding out about filling-in: a guide to perceptual completion for visual science and the philosophy of perception’, Behavioral and Brain Sciences, vol. 21, no. 6, pp. 723–48.

Petroni, A, Baguear, F & Della-Maggiore, V 2010, ‘Motor resonance may originate from sensorimotor experience’, Journal of Neurophysiology, vol. 104, no. 4, pp. 1867–71.

Pfeifer, JH, Iacoboni, M, Mazziotta, JC & Dapretto, M 2008, ‘Mirroring others' emotions relates to empathy and interpersonal competence in children’, NeuroImage, vol. 39, no. 4, pp. 2076–85.

Pinch, T & Bijsterveld, K 2004, ‘Sound Studies: new technologies and music’, Social Studies of Science, vol. 34, no. 5, pp. 635–48.

---- (eds) 2012, The Oxford handbook of sound studies, Oxford University Press, New York.

Pisters, P 2003, The matrix of visual culture: working with Deleuze in film theory, Stanford University Press, Stanford, California.

Place, UT 1956, ‘Is consciousness a brain process?’, British Journal of Psychology, vol. 47, pp. 44–50.

Plantinga, C 1993, ‘Affect, cognition, and the power of movies’, Post Script, vol. 13, no. 1, pp. 10–29.

---- 2009, Moving viewers: American film and the spectator's experience, University of California Press, Berkeley, CA.

Plantinga, CR 1999, ‘The scene of empathy and the human face on film’, in CR Plantinga & GM Smith (eds), Passionate views: film, cognition, and emotion, Johns Hopkins University Press, Baltimore, pp. 239– 55.

Plantinga, CR & Smith, GM (eds) 1999, Passionate views: film, cognition, and emotion, Johns Hopkins University Press, Baltimore.

Pockett, S 2004, ‘Does consciousness cause behaviour?’, Journal of Consciousness Studies, vol. 11, no. 2, pp. 23–40.

Popple, S & Kember, J 2004, Early cinema: from factory gate to dream factory, Wallflower, London.

Posner, MI, Nissen, MJ & Klein, M 1976, ‘Visual dominance: an information processing account of its origins and significance’, Psychological Review, vol. 83, no. 2, pp. 157–71.

Pourtois, G, de Gelder, B, Vroomen, J, Rossion, B & Crommelinck, M 2000, ‘The time-course of intermodal binding between seeing and hearing affective information’, NeuroReport, vol. 11, no. 6, pp. 1329–33.

Powell, A 2005, Deleuze and horror film, Edinburgh University Press, Edinburgh.

---- 2007, Deleuze, altered states and film, Edinburgh University Press, Edinburgh.

335 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Preston, SD & de Waal, FBM 2003, ‘Empathy: its ultimate and proximate bases’, Behavioral and Brain Sciences, vol. 25, no. 1, pp. 1–20.

Prince, G 2003, A dictionary of narratology, Revised edn, University of Nebraska, Lincoln; London.

Prince, S 1996, ‘True lies: perceptual realism, digital images, and film theory’, Film Quarterly, vol. 49, no. 3, pp. 27–37.

Prinz, J 2004, ‘Embodied emotions’, in RC Solomon (ed.), Thinking about feeling: contemporary philosophers on emotions, Oxford University Press, Oxford ; New York.

---- 2005, ‘A neurofunctional theory of consciousness’, in A Brook & K Akins (eds), Cognition and the brain: the philosophy and neuroscience movement, Cambridge University Press, Cambridge; New York, pp. 381–96.

Prinz, W 1984, ‘Modes of linkage between perception and action’, in W Prinz & A-F Sanders (eds), Cognition and motor processes, Springer, Berlin, pp. 185–93.

---- 1990, ‘A common-coding approach to perception and action’, in O Neumann & W Prinz (eds), Relationships between perception and action: current approaches, Springer-Verlag, Berlin, pp. 167–201.

---- 1997, ‘Perception and action planning’, European Journal of Cognitive Psychology, vol. 9, no. 2, pp. 129– 54.

---- 2003, ‘Experimental approaches to action’, in J Roessler & N Eilan (eds), Agency and self-awareness: issues in philosophy and psychology, Oxford University Press, New York, pp. 165–87.

Propp, V 1968, Morphology of the folktale, University of Texas Press, Austin.

Puckette, M 1996, Pure Data, San Francisco.

---- 1997, Max/MSP, Cycling '74, San Francisco.

Pudovkin, VI 1985 [1929], ‘Asynchronism as a principle of film sound’, in E Weis & J Belton (eds), Film sound: theory and practice, Columbia University Press, New York, pp. 86–91.

Pylyshyn, ZW 1973, ‘What the mind's eye tells the mind's brain: a critique of mental imagery’, Psychological bulletin, vol. 80, no. 1, pp. 1–24.

---- 1981, ‘The imagery debate: analogue media versus tacit knowledge’, Psychological Review, vol. 88, no. 1, pp. 16–45.

---- 1999, ‘Is vision continuous with cognition? The case for cognitive impenetrability of visual perception’, Behavioral and Brain Sciences, vol. 22, pp. 341–65.

---- 2002, ‘Mental imagery: in search of a theory’, Behavioral and Brain Sciences, vol. 25, no. 2, pp. 157–82.

---- 2003, ‘Return of the mental image: are there really pictures in the brain?’, Trends in Cognitive Sciences, vol. 7, no. 3, pp. 113–8.

Raij, T, McEvoy, L, Makela, JP & Hari, R 1997, ‘Human auditory cortex is activated by omissions of auditory stimuli’, Brain Research, vol. 745, no. 1-2, pp. 134–43.

Ramachandran, VS & Hirstein, W 1999, ‘The science of art: a neurological theory of aesthetic experience’, Journal of Consciousness Studies, vol. 6, no. 6/7, pp. 15–51.

Ramachandran, VS & Seckel, E 2011, ‘Neurology of visual aesthetics: Indian nymphs, modern art, and sexy beaks’, in AP Shimamura & SE Palmer (eds), Aesthetic science: connecting minds, brains, and experience, Oxford University Press, Oxford, pp. 375–89.

336 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Ramsaye, T 1926, A million and one nights: a history of the motion picture through 1925, Simon and Schuster, New York.

Rassell, A, Robinson, J, Verhagen, D, Pink, S, Redmond, S & Stadler, J 2016, ‘Seeing, sensing sound: eye- tracking soundscapes in Saving Private Ryan and Monsters, Inc.’, in CD Reinhard & CJ Olson (eds), Making sense of cinema: empirical studies into film spectators and spectatorship, Bloomsbury Academic, New York, pp. 139–64.

Reber, AS 1989, ‘Implicit learning and tacit knowledge’, Journal of Experimental Psychology: General, vol. 118, no. 3, pp. 219–35.

---- 1992, ‘The cognitive unconscious: an evolutionary perspective’, Consciousness and Cognition, vol. 1, no. 2, pp. 93–133.

Redfern, N 2011, ‘The road not taken in Film Studies’, Research into film: an empriical approach to film studies, 28 January 2016,

Reinoso Carvalho, F, Wang, Q, van Ee, R, Persoone, D & Spence, C 2017, ‘“Smooth operator”: music modulates the perceived creaminess, sweetness, and bitterness of chocolate’, Appetite, vol. 108, pp. 383–90.

Reisberg, D (ed.) 1992, Auditory imagery, Lawrence Erlbaum Associates, Hillsdale, N.J.

Repp, BH & Penel, A 2002, ‘Auditory dominance in temporal processing: new evidence from synchronization with simultaneous visual and auditory sequences’, Journal of Experimental Psychology: Human Perception and Performance, vol. 28, no. 5, pp. 1085–99.

Richardson, J, Gorbman, C & Vernallis, C (eds) 2013, The Oxford handbook of new audiovisual aesthetics, Oxford University Press, Oxford ; New York.

Richerson, PJ & Boyd, R 2000, ‘Built for speed: Pleistocene climate variation and the origin of human culture’, in F Tonneau & NS Thompson (eds), Evolution, culture, and behavior, Kluwer Academic/Plenum Publishers, New York, vol. 13, pp. 1–45.

Richerson, PJ & Boyd, R 2001, ‘Built for speed, not for comfort. Darwinian theory and human culture’, History and Philosophy of the Life Sciences, vol. 23, no. 3/4, pp. 423–63.

Richerson, PJ & Boyd, R 2005a, Not by genes alone: how culture transformed human evolution, University of Chicago Press, Chicago.

---- 2005b, ‘Nothing about culture makes sense except in the light of evolution’, in Not by genes alone: how culture transformed human evolution, University of Chicago Press, Chicago, pp. 237–57.

Richerson, PJ, Boyd, R & Henrich, J 2010, ‘Gene-culture coevolution in the age of genomics’, Proceedings of the National Academy of Sciences, vol. 107, no. Supplement 2, pp. 8985–92.

Richter, T, Appel, M & Calio, F 2013, ‘Stories can influence the self-concept’, Social Influence, vol. 9, no. 3, pp. 172–88.

Riva, G, Waterworth, JA & Waterworth, EL 2004, ‘The layers of presence: a bio-cultural approach to understanding presence in natural and mediated environments’, CyberPsychology & Behavior, vol. 7, no. 4, pp. 402–16.

Riva, G, Mantovani, F, Capideville, CS, Preziosa, A, Morganti, F, Villani, D, Gaggioli, A, Botella, C & Alcañiz, M 2007, ‘Affective interactions using virtual reality: the link between presence and emotions’, CyberPsychology & Behavior, vol. 10, no. 1, pp. 45–56.

Rizzolatti, G & Arbib, MA 1998, ‘Language within our grasp’, Trends in Neurosciences, vol. 21, no. 5, pp. 188– 94.

Rizzolatti, G & Fabbri-Destro, M 2010, ‘Mirror neuron mechanism’, in FK George, M Michel Le & FT Richard (eds), Encyclopedia of Behavioral Neuroscience, Academic Press, Oxford, pp. 240–9.

Rizzolatti, G, Fogassi, L & Gallese, V 2001, ‘Neurophysiological mechanisms underlying the understanding and imitation of action’, Nature Reviews Neuroscience, vol. 2, no. 9, pp. 661–70.

Robinson, CW & Sloutsky, VM 2004, ‘Auditory dominance and its change in the course of development’, Child Development, vol. 75, no. 5, pp. 1387–401.

---- 2007, ‘Visual processing speed: effects of auditory input on visual processing’, Developmental Science, vol. 10, no. 6, pp. 734–40.

Rockwell, T 2008, ‘Dynamic empathy: a new formulation for the simulation theory of mind reading’, Cognitive Systems Research, vol. 9, no. 1-2, pp. 52–63.

Rodowick, DN 1997, Gilles Deleuze's time machine, Duke University Press, Durham, NC.

---- 2010, Afterimages of Gilles Deleuze's film philosophy, University of Minnesota Press, Minneapolis.

Rohrer, T 2005, ‘Image schemata in the brain’, in B Hampe & JE Grady (eds), From perception to meaning: image schemas in cognitive linguistics, Mouton de Gruyter, Berlin; New York, pp. 165–96.

---- 2011, ‘Speaking without thinking: embodiment, speech technology and social signal processing’, in A Esposito, A Esposito, R Martone, V Müller & G Scarpetta (eds), Toward autonomous, adaptive, and context-aware multimodal interfaces. Theoretical and practical issues, Springer-Verlag, Berlin; Heidelberg, vol. 6456, pp. 8–22.

Rosenbaum, J 1978, ‘Sound thinking’, Film Comment, vol. 14, no. 5, pp. 38–41.

Rosenthal, DM 1986, ‘Two concepts of consciousness’, Philosophical Studies, vol. 49, no. 3, pp. 329–59.

---- 1990, ‘A theory of consciousness’, in N Block, O Flanagan & G Güzeldere (eds), The nature of consciousness: philosophical debates, MIT Press, Cambridge; London, pp. 729–53.

Ross, A (ed.) 1996, Science wars, Duke University Press, Durham.

Rothman, L 2014, ‘Game of Thrones breaks piracy record’, Time, 15 April 2014.

Roudaia, E, Sekuler, A, Bennett, P & Sekuler, R 2013, ‘Aging and audio-visual and multi-cue integration in motion’, Frontiers in Psychology, vol. 4, Article 267, pp. 1–14, doi 10.3389/fpsyg.2013.00267, viewed 23 May 2013, .

Russell, JA 2003, ‘Core affect and the psychological construction of emotion’, Psychological Review, vol. 110, no. 1, pp. 145–72.

---- 2005, ‘Emotion in human consciousness is built on core affect’, Journal of Consciousness Studies, vol. 8, no. 10, pp. 26–42.

---- 2009, ‘Emotion, core affect, and psychological construction’, Cognition and Emotion, vol. 23, no. 7, pp. 1259–83.

---- 2012, ‘From a psychological constructionist perspective’, in RD Ellis & P Zachar (eds), Categorical versus dimensional models of affect: a seminar on the theories of Panksepp and Russell, John Benjamins Publishing Company, Amsterdam ; Philadelphia, pp. 79–118.

Russell, JA & Barrett, LF 1999, ‘Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant’, Journal of Personality and Social Psychology, vol. 76, no. 5, pp. 805–19.

338 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Russell, JA & Lemay, G 2004, ‘Emotion concepts’, in M Lewis & JM Haviland-Jones (eds), Handbook of emotions, 2nd edn, Guilford Press, New York, pp. 491–503.

Russolo, L 1913, L'arte dei rumori: manifesto Futurista, Something Else Press, Milan.

Rutherford, A 2003, ‘Cinema and embodied affect’, Senses of Cinema, vol. 25, Article viewed 18 August 2011, .

Ryan, M-L 2007, ‘Toward a definition of narrative’, in D Herman (ed.), The Cambridge companion to narrative, Cambridge University Press, Cambridge ; New York, pp. 22–35.

Saad, G 2012, ‘Nothing in popular culture makes sense except in the light of evolution’, Review of General Psychology, vol. 16, no. 2, pp. 109–20.

Sadaghiani, S, Maier, JX & Noppeney, U 2009, ‘Natural, metaphoric, and linguistic auditory direction signals have distinct influences on visual motion processing’, The Journal of Neuroscience, vol. 29, no. 20, pp. 6490–9.

Salimpoor, VN, Benovoy, M, Longo, G, Cooperstock, JR & Zatorre, RJ 2009, ‘The rewarding aspects of music listening are related to degree of emotional arousal’, PLoS ONE, vol. 4, no. 10, Article e7487, pp. 1–14, doi 10.1371/journal.pone.0007487, viewed 21 August 2014, .

Salt, B 1978, ‘Film form 1900-1906’, Sight and Sound, vol. 47, no. 3, pp. 149–53.

---- 1984, ‘What we can learn from the first twenty years of cinema’, Iris, vol. 2, no. 1, pp. 83–90.

Sarris, A 1998, "You ain't heard nothin' yet": the American talking film, history and memory, 1927-1949, Oxford University Press, New York ; Oxford.

Saver, JL & Damásio, AR 1991, ‘Preserved access and processing of social knowledge in a patient with acquired sociopathy due to ventromedial frontal damage’, Neuropsychologia, vol. 29, no. 12, pp. 1241–9.

Scalise Sugiyama, M 2001a, ‘Food, foragers, and folklore: the role of narrative in human subsistence’, Evolution and Human Behavior, vol. 22, no. 4, pp. 221–40.

---- 2001b, ‘Narrative theory and function: why evolution matters’, Philosophy and Literature, vol. 25, no. 2, pp. 233–50.

---- 2005, ‘Reverse-engineering narrative: evidence of special design’, in J Gottschall & DS Wilson (eds), The literary animal: evolution and the nature of narrative, Northwestern University Press, Evanston, IL., pp. 177–96.

---- 2008, ‘Information is the stuff of narrative’, Style, vol. 42, no. 2/3, pp. 254–60.

---- 2009, ‘The plot thickens: what children's stories tell us about mindreading’, Journal of Consciousness Studies, vol. 16, no. 6/8, pp. 94–117.

Schaeffer, P 1952, In search of a concrète music, University of California Press, Berkeley, CA.

---- 1966, Traité des objets musicaux, Le Seuil, Paris, France.

Schafer, RM 1977, The tuning of the world, Knopf, New York.

---- 1993, The soundscape: our sonic environment and the tuning of the world, Destiny Books, Rochester, VT.

---- (ed.) 1973, The music of the environment, Universal Edition, Wien.

Schatz, T 1993, ‘The New Hollywood’, in J Collins, H Radner & AP Collins (eds), Film theory goes to the movies, Routledge, New York, pp. 8–37.

---- 2009, ‘New Hollywood, new millennium’, in W Buckland (ed.), Film theory and contemporary Hollywood movies, Routledge, New York, pp. 19–46.

Scheele, B 2001, ‘Back from the grave. Reinstating the catharsis concept in the psychology of reception’, in D Schram & G Steen (eds), The psychology and sociology of literature. In honor of Elrud Ibsch, John Benjamins Publishing Company, Amsterdam, pp. 201–24.

Scheele, B & DuBois, F 2006, ‘Catharsis as a moral form of entertainment’, in J Bryant & P Vorderer (eds), Psychology of entertainment, Lawrence Erlbaum Associates, Mahwah, NJ, pp. 405–22.

Scheier, CR, Nijwahan, R & Shimojo, S 1999, ‘Sound alters visual temporal resolution’, Investigative Ophthalmology and Visual Science, vol. 40, no. 4, p. S4169.

Schell, J 2008a, ‘Experiences can be judged by their interest curves’, in The art of game design: a book of lenses, Elsevier/Morgan Kaufmann, Amsterdam; Boston, pp. 245–60.

---- 2008b, The art of game design: a book of lenses, Elsevier/Morgan Kaufmann, Amsterdam; Boston.

Schellekens, E 2012, ‘Explanatory dualism in empirical aesthetics: a new reading’, Journal of Consciousness Studies, vol. 19, no. 9-10, pp. 200–19.

Scherer, K 2001, ‘Appraisal considered as a process of multilevel sequential checking’, in KR Scherer, A Schorr & T Johnstone (eds), Appraisal processes in emotion: theory, methods, research, Oxford University Press, New York, pp. 92–120.

---- 2003, Why music does not produce basic emotions: a plea for a new approach to measuring emotional effects of music, paper presented at Stockholm Music Acoustics Conference SMAC-03, Royal Institute of Technology, Stockholm.

Scherer, KR 1984a, ‘Emotion as a multicomponent process: a model and some cross-cultural data’, in P Shaver (ed.), Review of Personality & Social Psychology, Sage, Beverly Hills, CA, vol. 5, pp. 37–63.

Scherer, KR 1984b, ‘On the nature and function of emotion: a component process approach’, in KR Scherer & P Ekman (eds), Approaches to emotion, Erlbaum, Hillsdale, NJ, pp. 293–317.

Scherer, KR 1995, ‘Plato’s legacy: relationships between cognition, emotion, and motivation’, Geneva Studies in Emotion and Communication, vol. 9, no. 1, pp. 1–7.

Schirmer, A 2011, ‘How emotions change time’, Frontiers in Integrative Neuroscience, vol. 5, Article 58, pp. 1– 6, doi 10.3389/fnint.2011.00058, viewed 5 October 2011, .

Schlesinger, J 1969, Midnight Cowboy, motion picture, USA.

Schlicht, T 2012, ‘Phenomenal consciousness, attention and accessibility’, Phenomenology and the Cognitive Sciences, vol. 11, no. 3, pp. 309–34.

Schnall, S 2014, ‘Are there basic metaphors?’, in MJ Landau, MD Robinson & BP Meier (eds), The power of metaphor: examining its influence on social life, American Psychological Association, Washington, DC, pp. 225–47.

Schoeller, F 2015, ‘Knowledge, curiosity and aesthetic chills’, Frontiers in Psychology, vol. 6, no. 1546, pp. 1– 3.

Scorsese, M 1973, Mean Streets, motion picture, USA.

Scott de Martinville, É-L 1860, Au Clair de la Lune, sound recording, France.

Scott, R 1979, Alien, motion picture, USA, UK.

Sebastián, MÁ 2014, ‘Not a HOT Dream’, in R Brown (ed.), Consciousness inside and out: phenomenology, neuroscience, and the nature of experience, Springer Netherlands, vol. 6, pp. 415–32.

Sedgwick, EK & Frank, A 1995, ‘Shame in the cybernetic fold: reading Silvan Tomkins’, Critical Inquiry, vol. 21, no. 2, pp. 496–522.

---- 2003, Touching feeling: affect, pedagogy, performativity, Series Q, Duke University Press, Durham, NC.

Seeley, WP 2006, ‘Naturalizing aesthetics: art and the cognitive neuroscience of vision’, Journal of Visual Art Practice, vol. 5, no. 3, pp. 195–213.

Sekuler, R, Sekuler, AB & Lau, R 1997, ‘Sound alters visual motion perception’, Nature, vol. 385, no. 6614, p. 308.

Seligman, S 2009, ‘Anchoring intersubjective models in recent advances in developmental psychology, cognitive neuroscience and parenting studies: introduction to papers by Trevarthen, Gallese, and Ammaniti & Trentini’, Psychoanalytic Dialogues: The International Journal of Relational Perspectives, vol. 19, no. 5, pp. 503 – 6.

Sergent, C & Rees, G 2007, ‘Conscious access overflows overt report’, Behavioral and Brain Sciences, vol. 30, no. 5/6, pp. 523–4.

Sergi, G 1998, ‘A cry in the dark: the role of post-classical film sound’, in S Neale & M Smith (eds), Contemporary Hollywood cinema, Routledge, London; New York, pp. 156–65.

---- 2003, ‘Blockbusting sound: the case of The Fugitive’, in J Stringer (ed.), Movie blockbusters, Routledge, London, pp. 141–52.

---- 2004, The Dolby era: film sound in contemporary Hollywood, Inside popular film., Manchester University Press, Manchester.

---- 2006, ‘In defense of vulgarity: the place of sound effects in the cinema’, Scope, no. 5, Article 2, pp. 1–13, viewed 20 May 2017, .

---- 2013, ‘Knocking at the door of cinematic artifice: Dolby Atmos, challenges and opportunities’, The New Soundtrack, vol. 3, no. 2, pp. 107–21.

Sestir, M & Green, MC 2010, ‘You are who you watch: identification and transportation effects on temporary self-concept’, Social Influence, vol. 5, no. 4, pp. 272–88.

Shams, L & Kim, R 2010, ‘Crossmodal influences on visual perception’, Physics of Life Reviews, vol. 7, no. 3, pp. 269–84.

Shams, L, Kamitani, Y & Shimojo, S 2002, ‘Visual illusion induced by sound’, Cognitive Brain Research, vol. 14, no. 1, pp. 147–52.

---- 2004, ‘Modulations of visual perception by sound’, in G Calvert, C Spence & BE Stein (eds), Handbook of multisensory processes, MIT Press, Cambridge, Mass., pp. 27–33.

Shapiro, LA 2007, ‘The embodied cognition research programme’, Philosophy Compass, vol. 2, no. 2, pp. 338– 46.

---- 2011, Embodied cognition, Routledge, New York.

Sharpsteen, B 1941, Dumbo, motion picture, USA.

Shaver, P, Schwartz, J, Kirson, D & O'Connor, C 1987, ‘Emotion knowledge: further exploration of a prototype approach’, Journal of Personality and Social Psychology, vol. 52, no. 6, pp. 1061–86.

341 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Shaviro, S 1993, The cinematic body, Theory out of bounds ; v. 2, University of Minnesota Press, Minneapolis.

---- 2008, ‘The cinematic body REDUX’, Parallax, vol. 14, no. 1, pp. 48–54.

---- 2010, Post cinematic affect, 0-Books, Winchester, UK ; Washington, USA.

---- 2016, ‘Post-continuity: an introduction’, in S Denson & J Leyda (eds), Post-cinema: theorizing 21st-century film, REFRAME Books, Falmer, pp. 51–64.

Sheets-Johnstone, M 1999a, The primacy of movement, John Benjamins Publishing Company, Amsterdam; Philadelphia.

---- 1999b, ‘Emotion and movement. A beginning empirical-phenomenological analysis of their relationship’, Journal of Consciousness Studies, vol. 6, no. 11-12, pp. 259–77.

---- 2010, ‘Thinking in movement: further analyses and validations’, in J Stewart, O Gapenne & EA Di Paolo (eds), Enaction: towards a new paradigm for cognitive science, MIT Press, Cambridge, MA, pp. 165– 83.

Shephard, T & Leonard, A (eds) 2014, The Routledge companion to music and visual culture, Routledge, New York ; London.

Shimamura, AP 2011, ‘Toward a science of aesthetics: issues and ideas’, in AP Shimamura & SE Palmer (eds), Aesthetic science: connecting minds, brains, and experience, Oxford University Press, Oxford, pp. 3– 28.

---- 2013a, ‘Psychocinematics: issues and directions’, in AP Shimamura (ed.), Psychocinematics: exploring cognition at the movies, Oxford University Press, Oxford; New York, pp. 1–26.

---- (ed.) 2013b, Psychocinematics: exploring cognition at the movies, Oxford University Press, Oxford; New York.

Shimamura, AP & Palmer, SE (eds) 2011, Aesthetic science: connecting minds, brains, and experience, Oxford University Press, Oxford.

Shimojo, S & Shams, L 2001, ‘Sensory modalities are not separate modalities: plasticity and interactions’, Current Opinion in Neurobiology, vol. 11, no. 4, pp. 505–9.

Shimojo, S, Scheier, C, Nijhawan, R, Shams, L, Kamitani, Y & Watanabe, K 2001, ‘Beyond perceptual modality: auditory effects on visual perception’, Acoustical Science and Technology, vol. 22, no. 2, pp. 61–7.

Shweder, RA 1994, ‘“You’re not sick, you’re just in love”: emotion as an interpretive system’, in P Ekman & RJ Davidson (eds), The nature of emotion: fundamental questions, Oxford University Press, New York, pp. 32–44.

Sider, L, Freeman, D & Sider, J (eds) 2003, Soundscape: the School of Sound lectures, 1998-2001, Wallflower Press, London.

Silverman, K 1988, The acoustic mirror: the female voice in psychoanalysis and cinema, Theories of representation and difference, Indiana University Press, Bloomington.

Sinnerbrink, R 2009, ‘Hugo Münsterberg’, in F Colman (ed.), Film, theory and philosophy: the key thinkers, McGill-Queen's University Press, Montréal, pp. 20–30.

---- 2012, ‘Stimmung: exploring the aesthetics of mood’, Screen, vol. 53, no. 2, pp. 148–63.

---- 2016, ‘Cinempathy: phenomenology, cognitivism, and moving images’, Contemporary Aesthetics, vol. 5, Article viewed 13 December 2019, .

Slingerland, E 2005, ‘Conceptual blending, somatic marking, and normativity: a case example from ancient Chinese’, Cognitive Linguistics, vol. 16, no. 3, pp. 557–84.

---- 2008a, What science offers the humanities: integrating body and culture, Cambridge University Press, New York.

---- 2008b, ‘Who's afraid of reductionism? The study of religion in the age of cognitive science’, Journal of the American Academy of Religion, vol. 76, no. 2, pp. 375–411.

---- 2009, ‘Neuroscience, theory of mind, and the status of human-level truth’, in VP Gay (ed.), Neuroscience and religion: brain, mind, self and soul, Lexington Books, New York, pp. 67–108.

---- 2012, ‘Mind-body dualism and the Two Cultures’, in E Slingerland & M Collard (eds), Creating consilience: integrating the sciences and the humanities, Oxford University Press, New York, pp. 74– 87.

---- 2014, ‘Toward a second wave of consilience in the cognitive scientific study of religion’, Journal of Cognitive Historiography, vol. 1, no. 1, pp. 121–30.

Slingerland, E & Collard, M 2011a, ‘Creating consilience: toward a Second Wave’, in E Slingerland & M Collard (eds), Creating consilience: integrating the Sciences and the Humanities, Oxford University Press, New York, pp. 3–40.

---- (eds) 2011b, Creating consilience: integrating the Sciences and the Humanities, Oxford University Press, New York.

Sloboda, JA 1991, ‘Music structure and emotional response: some empirical findings’, Psychology of Music, vol. 19, no. 2, pp. 110–20.

Sloutsky, VM 2010, ‘From perceptual categories to concepts: what develops?’, Cognitive Science, vol. 34, no. 7, pp. 1244–86.

Sloutsky, VM & Napolitano, AC 2003, ‘Is a picture worth a thousand words? Preference for auditory modality in young children’, Child Development, vol. 74, no. 3, pp. 822–33.

Sloutsky, VM & Robinson, C 2008, ‘The role of words and sounds in infants' visual processing: from overshadowing to attentional tuning’, Cognitive Science, vol. 32, no. 2, pp. 342–65.

Smart, JJC 1959, ‘Sensations and brain processes’, The Philosophical Review, vol. 68, no. 2, pp. 141–56.

Smith, GM 1999, ‘Local emotions, global moods, and film structure’, in CR Plantinga & GM Smith (eds), Passionate views: film, cognition, and emotion, Johns Hopkins University Press, Baltimore, pp. 103– 26.

---- 2003a, ‘The Mood-cue approach to filmic emotion’, in Film structure and the emotion system, Cambridge University Press, Cambridge, NY, pp. 41–64.

---- 2003b, Film structure and the emotion system, Cambridge University Press, Cambridge, NY.

---- 2003c, ‘An invitation to feel’, in Film structure and the emotion system, Cambridge University Press, Cambridge, N.Y., pp. 3–14.

Smith, J 1996, ‘Unheard Melodies? A critique of psychoanalytic theories of film music’, in D Bordwell & N Carroll (eds), Post-theory: reconstructing film studies, University of Wisconsin Press, Madison, pp. 230–47.

---- 1999, ‘Movie music as moving music: emotion, cognition, and the film score’, in CR Plantinga & GM Smith (eds), Passionate views: film, cognition, and emotion, Johns Hopkins University Press, Baltimore, pp. 146–67.

Smith, J 2008, ‘Tearing speech to pieces: voice technologies of the 1940s’, Music, Sound, and the Moving Image, vol. 2, no. 2, pp. 183–206.

Smith, J 2013, ‘The sound of intensified continuity’, in J Richardson, C Gorbman & C Vernallis (eds), The Oxford Handbook of New Audiovisual Aesthetics, Oxford University Press, Oxford ; New York, pp. 331–56.

Smith, M 1995, Engaging characters: fiction, emotion, and the cinema, Oxford University Press, New York, NY.

---- 1998, ‘Theses on the philosophy of Hollywood history’, in S Neale & M Smith (eds), Contemporary Hollywood cinema, Routledge, London; New York, pp. 3–20.

---- 2011, ‘Triangulating aesthetic experience’, in AP Shimamura & SE Palmer (eds), Aesthetic science: connecting minds, brains, and experience, Oxford University Press, Oxford, pp. 80–106.

---- 2017, Film, art, and the Third Culture: a naturalized aesthetics of film, Oxford, Oxford University Press.

Smith, TJ 2013, ‘Watching you watch movies: using eye tracking to inform cognitive film theory’, in AP Shimamura (ed.), Psychocinematics: exploring cognition at the movies, Oxford University Press, New York, pp. 165–91.

Smith, TJ, Levin, D & Cutting, JE 2012, ‘A window on reality: perceiving edited moving images’, Current Directions in Psychological Science, vol. 21, no. 2, pp. 107–13.

Snow, CP 1956, ‘The Two Cultures’, New Statesman, 6 October 1956.

---- 1959, The Two Cultures and the Scientific Revolution, Cambridge University Press, New York.

Snyder, Z 2011a, Sucker Punch (teaser), online multimedia, viewed 12 October 2011

---- 2011b, Sucker Punch (trailer), online multimedia, viewed 12 October 2011

---- 2011c, Sucker Punch, motion picture, USA, Canada.

Sobchack, V 1992, The address of the eye: a phenomenology of film experience, Princeton University Press, Princeton, N.J.

---- 1994, ‘The scene of the screen: envisioning cinematic and electronic “presence”’, in HU Gumbrecht & KL Pfeiffer (eds), Materialities of communication, Stanford University Press, Stanford, Calif., pp. 83–106.

---- 2004, Carnal thoughts: embodiment and moving image culture, University of California Press, Berkeley.

Sokal, AD & Bricmont, J 1998, Intellectual impostures: postmodern philosophers' abuse of science, Profile Books, London.

Solberg, RT 2014, ‘“Waiting for the bass to drop”: correlations between intense emotional experiences and production techniques in build-up and drop sections of Electronic Dance Music’, Dancecult: Journal of Electronic Dance Music Culture, vol. 6, no. 1, pp. 61–82.

Solms, M 2013, ‘The conscious id’, Neuropsychoanalysis, vol. 15, no. 1, pp. 5–19.

Solms, M & Panksepp, J 2012, ‘The “Id” knows more than the “Ego” admits: neuropsychoanalytic and primal consciousness perspectives on the interface between affective and cognitive neuroscience’, Brain Sciences vol. 2, no. 2, pp. 147–75.

344 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Song, G, Pellerin, D & Granjon, L 2013, ‘Different types of sounds influence gaze differently in videos’, Journal of Eye Movement Research, vol. 6, no. 4, pp. 1–13.

Sonnenschein, D 2001, Sound design: the expressive power of music, voice, and sound effects in cinema, Michael Wiese Productions, Studio City, CA.

Sontag, S 1966, ‘Against interpretation’, in S Sontag (ed.), Against interpretation, and other essays, Farrar, Straus & Giroux, N.Y., pp. 3–14.

The Sopranos, 1999–2007, television drama, Home Box Office, USA.

Sounding Out!, 2009–present, viewed 14 January 2019, .

Spadoni, R 2007, Uncanny bodies: the coming of sound film and the origins of the horror genre, University of California Press, Berkeley, Calif. ; London.

Spielberg, S 1975, Jaws, motion picture, USA.

---- 1977, Close Encounters of the Third Kind, motion picture, USA.

---- 1993, Jurassic Park, motion picture, USA.

---- 1998, Saving Private Ryan, motion picture, USA.

Spitz, A undated, Social Sound Design, website, viewed 19 January 2019 squirrelsatemynuts 2005, Mediocre film; amazing sound design and nude scene, International Movie Database (IMDb), viewed 25 May 2012, .

Stadler, J 2016, ‘Experiential realism and motion pictures: a neurophenomenological approach’, Studia Phaenomenologica, vol. 16, pp. 439–65.

Steen, FF & Owens, SA 2001, ‘Evolution's pedagogy: an adaptationist model of pretense and entertainment’, Journal of Cognition and Culture, vol. 1, no. 4, pp. 289–321.

Stein, BE, London, N, Wilkinson, LK & Price, DD 1996, ‘Enhancement of perceived visual intensity by auditory stimuli: a psychophysical analysis’, Journal of Cognitive Neuroscience, vol. 8, no. 6, pp. 497– 506.

Stein, NL & Trabasso, T 1992, ‘The organisation of emotional experience: creating links among emotion, thinking, language, and intentional action’, Cognition and Emotion, vol. 6, no. 3/4, pp. 225–44.

Stern, DN 1999, ‘Vitality contours: the temporal contour of feelings as a basic unit for constructing the infant’s social experience’, in P Rochat (ed.), Early social cognition: understanding others in the first months of life, Erlbaum, Mahwah, N.J., pp. 67–90.

Sterne, J 2003, The audible past: cultural origins of sound reproduction, Duke University Press, Durham.

---- (ed.) 2012, The Sound Studies reader, Routledge, New York.

Stetson, C, Fiesta, MP & Eagleman, DM 2007, ‘Does time really slow down during a frightening event?’, PLoS ONE, vol. 2, no. 12, pp. 1–3.

Stewart, J, Gapenne, O & Di Paolo, EA (eds) 2010, Enaction: toward a new paradigm for cognitive science, MIT Press, Cambridge, Mass.

Stokes, D 2009, ‘Aesthetics and cognitive science’, Philosophy Compass, vol. 4, no. 5, pp. 715–33.

Stork, M 2011a, Chaos cinema (part 3), online multimedia, viewed 3 December 2019

---- 2011b, Chaos cinema (part 1), online multimedia, viewed 3 December 2019

---- 2011c, Chaos cinema (part 2), online multimedia, viewed 3 December 2019

---- 2013, ‘Chaos cinema: assaultive action aesthetics’, Media Fields Journal: Critical Explorations in Media and Space, vol. 6, Article viewed 5 December 2019, .

Strauven, W 2006a, ‘Introduction to an attractive concept’, in W Strauven (ed.), The cinema of attractions reloaded, Amsterdam University Press, Amsterdam, pp. 11–27.

---- (ed.) 2006b, The cinema of attractions reloaded, Amsterdam University Press, Amsterdam.

Strawson, G 2004a, ‘Against narrativity’, Ratio, vol. 17, no. 4, pp. 428–52.

---- 2004b, ‘A fallacy of our age: not every life is a narrative’, The Times Literary Supplement, 15 October 2004, pp. 13–5

Streri, A 2012, ‘Crossmodal interactions in the human newborn: new answers to Molyneux's question’, in AJ Bremner, DJ Lewkowicz & C Spence (eds), Multisensory development, Oxford University Press, Oxford, pp. 88–112.

Suddendorf, T & Corballis, MC 1997, ‘Mental time travel and the evolution of the human mind’, Genetic, Social, and General Psychology Monographs, vol. 123, no. 2, pp. 133–67.

---- 2007, ‘The evolution of foresight: what is mental time travel, and is it unique to humans?’, Behavioral and Brain Sciences, vol. 30, no. 3, pp. 299–313.

Suddendorf, T, Addis, DR & Corballis, MC 2009, ‘Mental time travel and the shaping of the human mind’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, no. 1521, pp. 1317– 24.

Sundararajan, L 2000, ‘Background-mood in emotional creativity: a microanalysis’, Consciousness and Emotion, vol. 1, no. 2, pp. 227–43.

---- 2008, ‘The plot thickens – or not: protonarratives of emotions and the Chinese principle of savoring’, Journal of Humanistic Psychology, vol. 48, no. 2, pp. 243–63.

Szalóky, M 2002, ‘Sounding images in silent film: visual acoustics in Murnau's Sunrise’, Cinema Journal, vol. 41, no. 2, pp. 109–31.

Tait, C 1906, The story of the Kelly Gang Australia.

Tajadura-Jiménez, A 2008, ‘Embodied psychoacoustics: spatial and multisensory determinants of auditory- induced emotion’, unpublished doctoral disertation, Chalmers University of Technology, Göteborg, Sweden.

Tajadura-Jiménez, A, Väljamäe, A, Asutay, E & Västfjäll, D 2010, ‘Embodied auditory perception: the emotional impact of approaching and receding sound sources’, Emotion, vol. 10, no. 2, pp. 216–29.

Tajadura-Jiménez, A, Larsson, P, Väljamäe, A, Västfjäll, D & Kleiner, M 2010, ‘When room size matters: acoustic influences on emotional responses to sounds’, Emotion, vol. 10, no. 3, pp. 416–22.

Tajadura-Jiménez, A, Kitagawa, N, Väljamäe, A, Zampini, M, Murray, M & Spence, C 2009, ‘Auditory- somatosensory multisensory interactions are spatially modulated by stimulated body surface and acoustic spectra’, Neuropsychologia, vol. 47, no. 1, pp. 195–203.

Takeshima, Y & Gyoba, J 2013, ‘Changing pitch of sounds alters perceived visual motion trajectory’, Multisensory Research, vol. 26, no. 4, pp. 317–32.

Tal-Or, N & Cohen, J 2010, ‘Understanding audience involvement: conceptualizing and manipulating identification and transportation’, Poetics, vol. 38, no. 4, pp. 402–18.

Tallis, R 1999, ‘A critical dictionary of neuromythology’, in On the edge of certainty: philosophical explorations, Macmillan Press, Houndmills, Basingstoke, UK, pp. 71–126.

---- 2011, Aping mankind: neuromania, Darwinitis and the misrepresentation of humanity, Acumen Publishing, Durham.

Tan, ESH 1994a, ‘Story processing as an emotion episode’, in H van Oostendorp & RA Zwaan (eds), Naturalistic text comprehension, Ablex, Norwood, NJ, pp. 165–88.

---- 1994b, ‘Film-induced affect as a witness emotion’, Poetics, vol. 23, no. 1/2, pp. 7–32.

---- 1996, Emotion and the structure of narrative film: film as an emotion machine, Lawrence Erlbaum Associates, Mahwah, NJ.

---- 1999, Emotion and style as design, paper presented at First International Conference on Design and Emotion, Delft, Netherlands, 3-5 November.

---- 2000, ‘Emotion, art, and the humanities’, in M Lewis & JM Haviland-Jones (eds), Handbook of emotions, 2nd edn, Guilford Press, New York, pp. 116–34.

---- 2008, ‘Entertainment is emotion: the functional architecture of the entertainment experience’, Media Psychology, vol. 11, no. 1, pp. 28–51.

---- 2018, ‘A psychology of the film’, Palgrave Communications, vol. 4, no. 1, pp. 1–20.

Tan, S-L, Spackman, MP & Bezdek, MA 2007, ‘Viewers' interpretations of film characters' emotions: effects of presenting film music before or after a character is shown’, Music Perception, vol. 25, no. 2, pp. 135– 52.

Tasker, Y 2004, ‘Introduction: action and adventure cinema’, in Y Tasker (ed.), Action and adventure cinema, Routledge, London, pp. 1–13.

Tati, J 1958, Mon Oncle, motion picture, France.

Teramoto, W, Hidaka, S & Sugita, Y 2010, ‘Sounds move a static visual object’, PLoS One, vol. 5, no. 8, p. e12255.

Teramoto, W, Hidaka, S, Gyoba, J & Suzuki, Y 2010, ‘Auditory temporal cues can modulate visual representational momentum’, Attention, Perception & Psychophysics, vol. 72, no. 8, pp. 2215–26.

Teramoto, W, Manaka, Y, Hidaka, S, Sugita, Y, Miyauchi, R, Sakamoto, S, Gyoba, J, Iwaya, Y & Suzuki, Y 2010, ‘Visual motion perception induced by sounds in vertical plane’, Neuroscience Letters, vol. 479, no. 3, pp. 221–5.

Tettamanti, M, Buccino, G, Saccuman, MC, Gallese, V, Danna, M, Scifo, P, Fazio, F, Rizzolatti, G, Cappa, SF & Perani, D 2005, ‘Listening to action-related sentences activates fronto-parietal motor circuits’, Journal of Cognitive Neuroscience, vol. 17, no. 2, pp. 273–81.

Thanouli, E 2006, ‘Post-classical narration: a new paradigm in contemporary cinema’, New Review of Film and Television Studies, vol. 4, no. 3, pp. 183–96.

---- 2008, ‘To be or not to be post-classical’, in J Kooijman, P Pisters & W Strauven (eds), Mind the screen: media concepts according to Thomas Elsaesser, Amsterdam University Press, Amsterdam, pp. 218–28.

---- 2009, Post-classical cinema: an international poetics of film narration, Wallflower Press, London.

Thom, R 1999, ‘Designing a movie for sound’, Iris, vol. 27, pp. 9–20.

---- 2007, ‘Acoustics of the soul’, Offscreen, vol. 11, no. 8/9, Article 6, viewed 9 July 2017, .

---- 2009, ‘On sound designing: Cast Away’, The Soundtrack, vol. 2, no. 1, pp. 19–21.

---- 2011, ‘Screenwriting for sound’, The New Soundtrack, vol. 1, no. 2, pp. 103–12.

---- 2013, ‘Notes on sound design in contemporary animated films’, in J Richardson, C Gorbman & C Vernallis (eds), The Oxford Handbook of New Audiovisual Aesthetics, Oxford University Press, Oxford ; New York, pp. 227–32.

Thompson, E 2004, The soundscape of modernity: architectural acoustics and the culture of listening in America, 1900–1933, MIT Press, Cambridge, MA.

Thompson, E 2005, ‘Sensorimotor subjectivity and the enactive approach to experience’, Phenomenology and the Cognitive Sciences, vol. 4, no. 4, pp. 407–27.

---- 2007, Mind in life: biology, phenomenology, and the sciences of mind, Harvard University Press Cambridge, MA.

Thompson, E & Stapleton, M 2009, ‘Making sense of sense-making: reflections on enactive and extended mind theories’, Topoi, vol. 28, no. 1, pp. 23–30.

Thompson, K 1977, ‘The concept of cinematic excess’, Ciné-tracts, vol. 1, no. 2, pp. 54–63.

---- 1999, Storytelling in the new Hollywood: understanding classical narrative technique, Harvard University Press, Cambridge, MA.

Thrift, N 2004, ‘Intensities of feeling: towards a spatial politics of affect’, Geografiska Annaler: Series B, Human Geography, vol. 86, no. 1, pp. 57–78.

Thrift, NJ 2008, Non-representational theory: space, politics, affect, Routledge, London.

Thurlow, WR & Jack, CE 1973, ‘Certain determinants of the "ventriloquism effect"’, Perceptual and Motor Skills, vol. 36, no. 3, pp. 1171–84.

Tieber, C & Windisch, AK (eds) 2014, The sounds of silent films: new perspectives on history, theory and practice, Palgrave studies in audio-visual culture, Palgrave Macmillan, Basingstoke.

Tierney, A & Kraus, N 2013, ‘Neural responses to sounds presented on and off the beat of ecologically valid music’, Frontiers in Systems Neuroscience, vol. 7, Article 14, pp. 1–7, doi 10.3389/fnsys.2013.00014, viewed 29 June 2015, .

Tilley, A 1992, Plot snakes and the dynamics of narrative experience, University Press of Florida, Gainesville, FL.

Tomasello, M 1999, The cultural origins of human cognition, Harvard University Press, Cambridge, MA.

Tomasello, M, Kruger, AC & Ratner, HH 1993, ‘Cultural learning’, Behavioral and Brain Sciences, vol. 16, no. 3, pp. 495–511.

Tomasello, M, Carpenter, M, Call, J, Behne, T & Moll, H 2005, ‘Understanding and sharing intentions: the origins of cultural cognition’, Behavioral and Brain Sciences, vol. 28, no. 5, pp. 675–735.

348 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Tooby, J & Cosmides, L 1992, ‘The psychological foundations of culture’, in JH Barkow, L Cosmides & J Tooby (eds), The adapted mind: evolutionary psychology and the generation of culture, Oxford University Press, New York, pp. 19–136.

---- 2001, ‘Does beauty build adapted minds? Toward an evolutionary theory of aesthetics, fiction, and the arts’, SubStance, vol. 30, no. 94/95, pp. 6–27.

Trevarthen, C 2006, ‘First things first: infants make good use of the sympathetic rhythm of imitation, without reason or language’, Journal of Child Psychotherapy, vol. 31, no. 1, pp. 91–113.

---- 2008, ‘The musical art of infant conversation: narrating in the time of sympathetic experience, without rational interpretation, before words’, Musicae Scientiae, vol. 12, no. 15, pp. 15–46.

Trost, W, Frühholz, S, Schön, D, Labbé, C, Pichon, S, Grandjean, D & Vuilleumier, P 2014, ‘Getting the beat: entrainment of brain activity by musical rhythm and pleasantness’, NeuroImage, vol. 103, pp. 55–64.

Truax, B 1978, The World Soundscape Project's Handbook for acoustic ecology, World Soundscape Project, A.R.C. Publications, Vancouver, B.C.

---- 2008, ‘Soundscape composition as global music: electroacoustic music as soundscape’, Organised Sound, vol. 13, no. 2, pp. 103–9.

Truffaut, F 1978, Hitchcock, Paladin, London.

Trumpp, NM, Traub, F, Pulvermüller, F & Kiefer, M 2013, ‘Unconscious automatic brain activation of acoustic and action-related conceptual features during masked repetition priming’, Journal of Cognitive Neuroscience, vol. 26, no. 2, pp. 352–64.

Tsivian, Y 2008, ‘"What Is cinema?" An agnostic answer’, Critical Inquiry, vol. 34, no. 4, pp. 754–76.

Turner, M 1996, The literary mind, Oxford University Press, New York.

---- 2006a, ‘Compression and representation’, Language and Literature, vol. 15, no. 1, pp. 17–27.

---- 2006b, ‘The art of compression’, in M Turner (ed.), The artful mind: cognitive science and the riddle of human creativity, Oxford University Press, New York, pp. 93–113.

Turner, M & Fauconnier, G 1995, ‘Conceptual integration and formal expression’, Metaphor & Symbolic Activity, vol. 10, no. 3, pp. 183–204.

Tuuri, K & Eerola, T 2012, ‘Formulating a revised taxonomy for modes of listening’, Journal of New Music Research, vol. 41, no. 2, pp. 137–52.

Tuuri, K, Mustonen, M-S & Pirhonen, A 2007, ‘Same sound different meanings: a novel scheme for modes of listening’, Proceedings of Audio Mostly, pp. 13–8.

Tykwer, T, Wachowski, A & Wachowski, L 2012, Cloud Atlas, motion picture, Germany, USA, Hong Kong, Singapore.

Umiltà, MA, Kohler, E, Gallese, V, Fogassi, L, Fadiga, L, Keysers, C & Rizzolatti, G 2001, ‘I know what you are doing: a neurophysiological study’, Neuron, vol. 31, no. 1, pp. 155–65.

Väljamäe, A & Tajadura-Jiménez, A 2007, ‘Perceptual optimization of audio-visual media: moved by sound’, in JD Anderson & BF Anderson (eds), Narration and spectatorship in moving images, Cambridge Scholars Publishing, Newcastle, UK, pp. 210–21.

Väljamäe, A & Soto-Faraco, S 2008, ‘Filling-in visual motion with sounds’, Acta Psychologica, vol. 129, no. 2, pp. 249–54.

349 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Väljamäe, A, Västfjäll, D, Larsson, P & Kleiner, M 2008, ‘Perceived sound in mediated environments’, in F Biocca, W IJsselsteijn & J Freeman (eds), Immersed in media experiences: presence psychology and design (Handbook of Presence), Lawrence Erlbaum Associates.

Van der Burg, E, Olivers, CNL, Bronkhorst, AW & Theeuwes, J 2008, ‘Pip and pop: nonspatial auditory signals improve spatial visual search’, Journal of Experimental Psychology: Human Perception and Performance, vol. 34, no. 5, pp. 1053–65. van Wassenhove, V, Grant, KW & Poeppel, D 2007, ‘Temporal window of integration in auditory-visual speech perception’, Neuropsychologia, vol. 45, no. 3, pp. 598–607. van Wassenhove, V, Wittmann, M, Craig, AD & Paulus, MP 2011, ‘Psychological and neural mechanisms of subjective time dilation’, Frontiers in Neuroscience, vol. 5, Article 56, pp. 1–10, doi 10.3389/fnins.2011.00056, viewed 26 April 2011, .

Varela, FJ 1996, ‘Neurophenomenology: a methodological remedy for the hard problem’, Journal of Consciousness Studies, vol. 3, no. 4, pp. 330–49.

Varela, FJ & Depraz, N 2005, ‘At the source of time: valence and the constitutional dynamics of affect’, Journal of Consciousness Studies, vol. 12, no. 8/10, pp. 61–81.

Varela, FJ, Thompson, E & Rosch, E 1991, The embodied mind: cognitive science and human experience, MIT Press, Cambridge, Mass.

Västfjäll, D 2001, ‘Emotion induction through music: a review of the Musical Mood Induction Procedure’, Musicae Scientiae, vol. 5, no. 1, pp. 173–211.

---- 2003, ‘The subjective sense of presence, emotion recognition, and experienced emotions in auditory virtual environments’, CyberPsychology & Behavior, vol. 6, no. 2, pp. 181–8.

Västfjäll, D & Kleiner, M 2001, Emotion in product sound design, paper presented at Journes du Design Sonore, Paris, 20-21 March 2002.

Västfjäll, D, Larsson, P & Kleiner, M 2002, ‘Emotion and auditory virtual environments: affect-based judgments of music reproduced with virtual reverberation times’, CyberPsychology & Behavior, vol. 5, no. 1, pp. 19–32.

Verbinski, G 2002, The Ring, motion picture, USA.

Verma, N 2012, Theater of the mind: imagination, aesthetics, and American radio drama, University of Chicago Press, Chicago ; London.

Vernallis, C 2013, Unruly media: YouTube, music video, and the new digital cinema, Oxford University Press, Oxford; New York.

Vernallis, C, Herzog, A & Richardson, J (eds) 2013, The Oxford handbook of sound and image in digital media, Oxford University Press, Oxford ; New York.

Versace, R & Rose, M 2007, ‘The role of emotion in multimodal integration’, Current Psychology Letters, Behavior, Brain, and Cognition, vol. 21, no. 1, pp. 13–22.

Vertov, D 1929, Man with a Movie Camera, motion picture, USSR.

---- 1930, Enthusiasm: The Symphony of Donbass, motion picture, USSR.

---- 1984, ‘From Kino-Eye to Radio-Eye’, in A Michelson (ed.), Kino-eye: the writings of Dziga Vertov, University of California Press, Berkeley, CA, pp. 85–92.

350 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Vickers, E 2010, The loudness war: background, speculation, and recommendations, paper presented at AES 129th Convention, San Francisco, CA, USA, 4–7 November, 2010.

Vickers, P 2012, ‘Ways of listening and modes of being: electroacoustic auditory display’, Journal of Sonic Studies, vol. 2, no. 1, Article viewed 7 August 2019, .

Villeneuve, D 2016, Arrival, motion picture, USA.

Vines, BW, Nuzzo, RL & Levitin, DJ 2005, ‘Analyzing temporal dynamics in music’, Music Perception, vol. 23, no. 2, pp. 137–52.

Visch, VT 2004, ‘Moved by movements: how character movements cue us to form specific genre and affective impressions’, in M Rauterberg (ed.), Entertainment Computing – ICEC 2004, Springer Berlin/Heidelberg, vol. 3166, pp. 49–72.

Visch, VT & Tan, ESH 2007, ‘Effect of film velocity on genre recognition’, Media Psychology, vol. 9, no. 1, pp. 59–75.

---- 2008, ‘Narrative versus style: effect of genre-typical events versus genre-typical filmic realizations on film viewers' genre recognition’, Poetics, vol. 36, no. 4, pp. 301–15.

---- 2009, ‘Categorizing moving objects into film genres: the effect of animacy attribution, emotional response, and the deviation from non-fiction’, Cognition, vol. 110, no. 2, pp. 265–72.

Visch, VT, Tan, ESH & Molenaar, D 2010, ‘The emotional and cognitive effect of immersion in film viewing’, Cognition and Emotion, vol. 24, no. 8, pp. 1439–45.

Voegelin, S 2010, Listening to noise and silence: towards a philosophy of sound art, Continuum, New York.

Vogeley, K & Roepstorff, A 2009, ‘Contextualising culture and social cognition’, Trends in Cognitive Sciences, vol. 13, no. 12, pp. 511–6.

Voisin, J, Bidet-Caulet, A, Bertrand, O & Fonlupt, P 2006, ‘Listening in silence activates auditory areas: a functional magnetic resonance imaging study’, The Journal of Neuroscience, vol. 26, no. 1, pp. 273–8.

Vorderer, P 2001, ‘It's all entertainment - sure. But what exactly is entertainment? Communication research, media psychology, and the explanation of entertainment experiences’, Poetics, vol. 29, no. 4-5, pp. 247–61.

Vorderer, P & Hartmann, T 2009, ‘Entertainment and enjoyment as media effects’, in J Bryant & MB Oliver (eds), Media effects: advances in theory and research, 3rd edn, Routledge, New York, pp. 532–50.

Vorderer, P, Klimmt, C & Ritterfeld, U 2004, ‘Enjoyment: at the heart of media entertainment’, Communication Theory, vol. 14, no. 4, pp. 388–408.

Vroomen, J & de Gelder, B 2000, ‘Sound enhances visual perception: cross-modal effects of auditory organization on vision’, Journal of Experimental Psychology Human Perception and Performance, vol. 26, no. 5, pp. 1583–90.

Vroomen, J & de Gelder, B 2004, ‘Temporal ventriloquism: sound modulates the flash-lag effect’, Journal of Experimental Psychology: Human Perception and Performance, vol. 30, no. 3 pp. 513–8

Vroomen, J, Driver, J & de Gelder, B 2001, ‘Is cross-modal integration of emotional expressions independent of attentional resources?’, Cognitive, Affective, and Behavioral Neuroscience, vol. 1, no. 4, pp. 382–7.

Wachowski, A & Wachowski, L 1999, The Matrix, motion picture, USA, Australia.

Walsh, R 2011, ‘The common basis of narrative and music: somatic, social, and affective foundations’, StoryWorlds: A Journal of Narrative Studies, vol. 3, pp. 49–71.

351 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Wang, Q, Keller, S & Spence, C 2017, ‘Sounds spicy: enhancing the evaluation of piquancy by means of a customised crossmodally congruent soundtrack’, Food Quality and Preference, vol. 58, pp. 1–9.

Wang, Y & Bargh, JA 2016, ‘Neural reuse leads to associative connections between concrete (physical) and abstract (social) concepts and motives’, Behavioral and Brain Sciences, vol. 39, no. e134.

Warburton, D 2008, ‘Interview with Michel Chion’, Paris Transatlantic Media, Article 6 June 2008, viewed 4 February 2015, .

Ward, J 2004, ‘Emotionally mediated synaesthesia’, Cognitive Neuropsychology, vol. 21, no. 7, pp. 761–72.

---- 2013, ‘Synesthesia’, Annual Review of Psychology, vol. 64, no. 1, pp. 49–75.

Ward, J & Mattingley, JB 2006, ‘Synaesthesia: an overview of contemporary findings and controversies’, Cortex, vol. 42, no. 2, pp. 129–36.

Ward, MS 2010, ‘Voice, videogames, and the technologies of immersion’, in N Neumark, R Gibson & TV Leeuwen (eds), Voice: vocal aesthetics in digital arts and media, MIT Press, Cambridge, Mass., pp. 267–79.

---- 2013, Art in noise: an embodied simulation account of cinematic sound design, paper presented at Annual conference of the Society for Cognitive Studies of the Moving Image, Universität der Künste, Berlin, 12-15 June 2013.

---- 2015, ‘Art in noise: an embodied simulation account of cinematic sound design’, in M Coëgnarts & P Kravanja (eds), Embodied cognition and cinema, Leuven University Press, Leuven, Belgium, pp. 155– 86.

Ward, MS & Leung, L 2008, ‘Beyond the visual: applying cinematic sound design to the online environment’, in L Leung (ed.), Digital experience design: ideas, industries, interaction, Intellect, Bristol, UK, pp. 81– 91.

Watanabe, K & Shimojo, S 2001, ‘When sound affects vision: effects of auditory grouping on visual motion perception’, Psychological Science, vol. 12, no. 2, pp. 109–16.

Weeks, R, Horwitz, B, Aziz-Sultan, A, Tian, B, Wessinger, CM, Cohen, LG, Hallett, M & Rauschecker, JP 2000, ‘A positron emission tomographic study of auditory localization in the congenitally blind’, The Journal of Neuroscience, vol. 20, no. 7, p. 2664.

Wegner, DM 2002, The illusion of conscious will, MIT Press, Cambridge, Mass. ; London.

---- 2003, ‘The mind's best trick: how we experience conscious will’, Trends in Cognitive Sciences, vol. 7, no. 2, pp. 65–9.

Wegner, DM & Bargh, JA 1998, ‘Control and automaticity in social life’, in DT Gilbert, ST Fiske & G Lindzey (eds), The handbook of social psychology, 4th edn, McGraw-Hill, New York, NY, vol. 1, pp. 446–96.

Weis, E & Belton, J (eds) 1985, Film sound: theory and practice, Columbia University Press, New York.

Welles, O 1938, The War of the Worlds, radio drama, CBS Radio, USA.

---- 1941, Citizen Kane, motion picture, USA.

Wheeler, M & Clark, A 2008, ‘Culture, embodiment and genes: unravelling the triple helix’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 363, no. 1509, pp. 3563–75.

Whittington, W 2007, Sound design and science fiction, University of Texas Press, Austin, Texas.

---- 2009, ‘Sound design in New Hollywood cinema’, in G Harper, R Doughty & J Eisentraut (eds), Sound and music in film and visual media: a critical overview, Continuum, London, pp. 555–68.

---- 2010, ‘Film, a sound art (review)’, Music, Sound, and the Moving Image, vol. 4, no. 2, pp. 253–60.

---- 2013, ‘Lost in sensation: reevaluating the role of cinematic sound in the digital age’, in C Vernallis, A Herzog & J Richardson (eds), The Oxford handbook of sound and image in digital media, Oxford University Press, Oxford ; New York, pp. 61–74.

Wiens, S 2005, ‘Interoception in emotional experience’, Current Opinion in Neurology, vol. 18, no. 4, pp. 442– 7.

Williams, L 1989, Hard core: power, pleasure, and the “frenzy of the visible", University of California Press, Berkeley, CA.

---- 1991, ‘Film bodies: gender, genre, and excess’, Film Quarterly, vol. 44, no. 4, pp. 2–13.

Wilson, EO 1998, Consilience: the unity of knowledge, Knopf, New York.

Wilson, M 2002, ‘Six views of embodied cognition’, Psychonomic Bulletin & Review, vol. 9, no. 4, pp. 625–36.

---- 2008, ‘How did we get from there to here? An evolutionary perspective on embodied cognition’, in C Paco & G Antoni (eds), Handbook of cognitive science, Elsevier, San Diego, pp. 373–93.

---- 2010, ‘The re-tooled mind: how culture re-engineers cognition’, Social Cognitive and Affective Neuroscience, vol. 5, no. 2/3, pp. 180–7.

Wilson, TD 2002, Strangers to ourselves: discovering the adaptive unconscious, Belknap Press, Cambridge, Mass.

Wimsatt, W & Beardsley, MC 1949, ‘The affective fallacy’, The Sewanee Review, vol. 57, no. 1, pp. 31–55.

---- 1954, ‘The affective fallacy’, in The verbal icon: studies in the meaning of poetry, University Press of Kentucky, Lexington, KY, pp. 21–39.

Winkielman, P 2010, ‘Bob Zajonc and the unconscious emotion’, Emotion Review, vol. 2, no. 4, pp. 353–62.

Winkielman, P & Berridge, KC 2004, ‘Unconscious emotion’, Current Directions in Psychological Science, vol. 13, no. 3, pp. 120–3.

Winkielman, P & Schooler, JW 2008, ‘Unconscious, conscious, and metaconscious in social cognition’, in F Strack & J Foerster (eds), Social cognition: the basis of human interaction, Psychology Press, Philadelphia, PA, pp. 49–69.

---- 2011, ‘Splitting consciousness: unconscious, conscious, and metaconscious processes in social cognition’, European Review of Social Psychology, vol. 22, no. 1, pp. 1–35.

Winkielman, P, Berridge, KC & Wilbarger, JL 2005, ‘Unconscious affective reactions to masked happy versus angry faces influence consumption behavior and judgments of value’, Personality and Social Psychology Bulletin, vol. 31, no. 1, pp. 121–35.

Winkielman, P, Niedenthal, PM & Oberman, L 2008, ‘The embodied emotional mind’, in GR Semin & ER Smith (eds), Embodied grounding: social, cognitive, affective, and neuroscientific approaches, Cambridge University Press, New York, pp. 263–88.

Winkielman, P, McIntosh, DN & Oberman, L 2009, ‘Embodied and disembodied emotion processing: learning from and about typical and autistic individuals’, Emotion Review, vol. 1, no. 2, pp. 178–90.

Winkielman, P, Niedenthal, PM & Oberman, LM 2009, ‘Embodied perspective on emotion-cognition interactions’, in JA Pineda (ed.), Mirror neuron systems: the role of mirroring processes in social cognition, Humana Press, New York, pp. 235–57.

Wojciehowski, HC & Gallese, V 2011, ‘How stories make us feel: toward an embodied narratology’, California Italian Studies Journal, vol. 2, no. 1, Article 38, viewed 21 March 2012, .

Wöllner, C (ed.) 2017, Body, sound and space in music and beyond: multimodal explorations, Routledge, London; New York.

Wood, A 2002, ‘Timespaces in spectacular cinema: crossing the great divide of spectacle versus narrative’, Screen, vol. 43, no. 4, pp. 370–86.

Woodruff, B 1997, My Heart Will Go On, music video, USA.

Wooley, BE & March, DS 2014, Exploring the influence of audio in directing visual attention during dynamic content, paper presented at ETRA '14: Symposium on Eye Tracking Research and Applications, Safety Harbor, Florida, 26–28 March 2014.

Wright, B 2013, ‘What do we hear? The pluralism of sound design in Hollywood sound production’, The New Soundtrack, vol. 3, no. 2, pp. 137–57.

---- 2014, ‘Footsteps with character: the art and craft of Foley’, Screen, vol. 55, no. 2, pp. 204–20.

Wyatt, J 1994, High concept: movies and marketing in Hollywood, Texas film studies series., University of Texas Press, Austin.

Wyer, RS, Adaval, R & Colcombe, SJ 2002, ‘Narrative-based representations of social knowledge: their construction and use in comprehension, memory and judgment’, in MP Zanna (ed.), Advances in experimental social psychology, Academic Press, Boston, MA, vol. 34, pp. 133–99.

Yamada, Y & Kawabe, T 2011, ‘Emotion colors time perception unconsciously’, Consciousness and Cognition, vol. 20, no. 4, pp. 1835–41.

Yeh, W & Barsalou, LW 2006, ‘The situated nature of concepts’, The American Journal of Psychology, vol. 119, no. 3, pp. 349–84.

Yewdall, DL 2003, Practical art of motion picture sound, 2nd edn, Focal Press, Oxford ; Boston.

Young, L & Koenigs, M 2007, ‘Investigating emotion in moral cognition: a review of evidence from functional neuroimaging and neuropsychology’, British Medical Bulletin, vol. 84, no. 1, pp. 69–79.

Zacks, JM & Tversky, B 2001, ‘Event structure in perception and conception’, Psychological Bulletin, vol. 127, no. 1, pp. 3–21.

Zacks, JM & Swallow, KM 2007, ‘Event segmentation’, Current Directions in Psychological Science, vol. 16, no. 2, pp. 80–4.

Zacks, JM & Magliano, JP 2011, ‘Film, narrative, and cognitive neuroscience’, in F Bacci & D Melcher (eds), Art and the senses, Oxford University Press, Oxford ; New York, pp. 435–54.

Zacks, JM, Speer, NK, Swallow, KM & Maley, CJ 2010, ‘The brain's cutting-room floor: segmentation of narrative cinema’, Frontiers in Human Neuroscience, vol. 4, Article 168, pp. 1–15, doi 10.3389/fnhum.2010.00168, viewed 24 April 2015, .

Zacks, JM, Speer, NK, Swallow, KM, Braver, TS & Reynolds, JR 2007, ‘Event perception: a mind-brain perspective’, Psychological Bulletin, vol. 133, no. 2, pp. 273–93.

354 | ©2017 Mark Shelton Ward. PhD Thesis, Faculty of Arts and Design, University of Canberra. Zacks, JM, Braver, TS, Sheridan, MA, Donaldson, DI, Snyder, AZ, Ollinger, JM, Buckner, RL & Raichle, ME 2001, ‘Human brain activity time-locked to perceptual event boundaries’, Nature Neuroscience, vol. 4, no. 6, pp. 651–5.

Zahavi, D 2007, ‘Self and other: the limits of narrative understanding’, in DD Hutto (ed.), Narrative and understanding persons, Cambridge University Press, Cambridge, pp. 179–201.

Zajonc, RB 1980, ‘Feeling and thinking: preferences need no inferences’, American Psychologist, vol. 35, no. 2, pp. 151–75.

---- 1984, ‘On the primacy of affect’, American Psychologist, vol. 39, no. 2, pp. 117–23.

---- 2000, ‘Feeling and thinking: closing the debate over the independence of affect’, in JP Forgas (ed.), Feeling and thinking: the role of affect in social cognition, Cambridge University Press, Cambridge, UK; New York; Paris, pp. 31–58.

---- 2001, ‘Mere exposure: a gateway to the subliminal’, Current Directions in Psychological Science, vol. 10, no. 6, p. 224.

Zakharine, D & Meise, N (eds) 2013, Electrified voices: medial, socio-historical and cultural aspects of voice transfer, V&R Unipress, Göttingen.

Zeelenberg, R & Bocanegra, BR 2010, ‘Auditory emotional cues enhance visual perception’, Cognition, vol. 115, no. 1, pp. 202–6.

Zeelenberg, R, Wagenmakers, E-J & Rotteveel, M 2006, ‘The impact of emotion on perception: bias or enhanced processing?’, Psychological Science, vol. 17, no. 4, pp. 287–91.

Zeki, S 1999a, ‘Art and the brain’, Journal of Consciousness Studies, vol. 6, no. 6-7, pp. 76–96.

---- 1999b, Inner vision: an exploration of art and the brain, Oxford University Press, Oxford.

---- 2009, Splendors and miseries of the brain: love, creativity, and the quest for human happiness, Wiley- Blackwell, Oxford.

Zillmann, D 1971, ‘Excitation transfer in communication-mediated aggressive behavior’, Journal of Experimental Social Psychology, vol. 7, no. 4, pp. 419–34.

---- 1983, ‘Transfer of excitation in emotional behavior’, in JT Cacioppo & RE Petty (eds), Social psychophysiology: a sourcebook, Guilford Press, New York, pp. 215–40.

---- 2000, ‘The coming of media entertainment’, in D Zillmann & P Vorderer (eds), Media entertainment: the psychology of its appeal, Lawrence Erlbaum Associates Publishers, Mahwah, N.J., pp. 1–20.

Žižek, S 2002, ‘Cultural Studies versus the "Third Culture"’, The South Atlantic Quarterly vol. 101, no. 1, pp. 19–32.

Zunshine, L 2008a, Strange concepts and the stories they make possible: cognition, culture, narrative, Johns Hopkins University Press, Baltimore, MD.

---- 2008b, ‘Theory of mind and fictions of embodied transparency’, Narrative, vol. 16, no. 1, pp. 65–92.

---- (ed.) 2010, Introduction to cognitive cultural studies, Johns Hopkins University Press, Baltimore.

Zwaan, RA 1999, ‘Embodied cognition, perceptual symbols, and situation models’, Discourse Processes, vol. 28, no. 1, pp. 81–8.