The Constituent Aspects of Synchresis: a Practice-Led Examination of Audio-Visual Synchronization in Key Sites of Production
The constituent aspects of synchresis: A practice-led examination of audio-visual synchronization in key sites of production
Hugo Presser
Submitted in fulfilment of the requirements for the degree of
Doctor of Philosophy
Swinburne University of Technology
2021
Abstract
In his seminal work Audio-Vision, Chion (1994) uses the term synchresis to refer to instances of audio- visual synchronization in film and video production. From his examination of audio-visual practice and the various exercises to discuss and produce synchronization, there have subsequently been few efforts to identify the constituent elements of form and the characteristic uses of synchresis in screen- production. Using Chion’s (1994) discussion as a framework, this research project has sought to deconstruct and propose a taxonomy of the constituent elements of a single instance of synchresis.
With the aim of deconstructing audio-visual synchronism and forging new approaches to synchresis, the methods used in this project contribute a unique research undertaking in the form of a practice-led method that is supported by an examination of synchresis in music video and commercial advertising production. This submission is comprised of two parts: a written component herein, alongside an artefact component of original music and music video that has been created in dialogue with the written submission. To distinguish clearly between the varied approaches to synchresis in the artefact submission, the music and music videos are submitted under monikers that are each played by me, with a varying approach to the synchresis in production: It's the Delmer Brothers!, Pap, Peter Bowman Jr, The Fed-Up Kids.
As a means of observing and identifying the ways in which synchresis occurs in media production, the practice-led component of music and music videos is submitted alongside the identification and supporting examination of commercial advertising and music video production. These discussions are interwoven with the explication of my own audio-visual practice, as a means of articulating each as a response to the identification of an instance of synchresis in existing media production. Although I have suggested in the conclusion that the aim of identifying and presenting a taxonomy of audio-visual synchronism was not achieved, the outcomes are valuable in articulating how and why it is problematic to deconstruct and categorise audio-visual synchronism rigorously. Given this, the conclusions that have been obtained from this research project are problematic, though they suggest how the process of obtaining a taxonomy is fraught with inconsistencies and of the difficulties of rigorously deconstructing and creating a taxonomy for audio-visual synchronism in media production. Acknowledgements
Firstly, I would like to express my gratitude to my primary supervisor throughout this process, Doctor James Verdon, for bearing with my constant rearrangement of the exegesis and artefact, and through the extension of my candidature in light of various setbacks.
In addition to my primary supervisors, I would like to thank and acknowledge the assistance of my various secondary supervisors, Associate Professor Deirdre Barron and Doctor Ramon Lobato. Without their help, scrutiny, patience and encouragement, the accompanying written component would likely still be incomplete.
Besides my primary and secondary supervisors, I would like to thank the various panel members: Associate Professor Simone Taffe, Doctor Carolyn Beasley, Doctor Steven Conway and Doctor Mark Freeman who each provided feedback and reviewed my work across the lengthy course of the candidature.
I would like to extend my love and thanks to my family, my mother Christine and father Lutz, my sister Niele for their love and support. Foremost, to Jaime for her love, support, and endless patience in bearing with me through the submission. Declaration
I Hugo Presser do declare that the thesis contains no material which has been accepted for the award to the candidate of any degree or diploma, except where due reference is made in the text of the thesis.
That to the best of my knowledge this thesis contains no material previously published or written by another person except where due reference is made in the text of the thesis, and that where the work in this thesis is based on joint research or publications that the relative contributions of the respective workers or authors has been disclosed.
20/06/2019
Hugo Presser Date
This thesis has been edited by Professor Margaret Zeegers. This editing has addressed only the style and/or grammar of the thesis and not its substantive content. Table of Contents
Table of Contents ...... i
List of Figures ...... i
A List of Titles Comprising the Artefact Submission ...... iv
An Overview of the Project ...... 8
A Discussion of the Screen-Production Research Methodology ...... 10
The Screen-Production Framework of the Project ...... 12
A Brief Overview of Audio-Visual Synchronization ...... 19
Animation and Synchronous Sound ...... 25
The Classifications of the Taxonomy in the Project ...... 29
Points of Audio-Visual Synchronization ...... 33
The Synchretic Envelope of a Synch-Point ...... 34
Synchretic Parameters and Operations ...... 35
Music Video as a Site of Audio-Visual Synchronisation ...... 37
Commercial Advertising as a Site of Audio-Visual Synchronisation ...... 40
Medium-Specific Noise in the Production of the Artefact Submission ...... 44
The Working Processes Comprising the Artefact Component ...... 48
It’s the Delmer Brothers!—Commercials 1975 – 1981 ...... 48
The Fed-Up Kids—Campus Slump OST (1981) ...... 49
Peter Bowman Jr.—City Nights (1984) ...... 49
Pap—Self-Titled (2019) ...... 50
i
Omitted Compilation ...... 51
A Statement Addressing the Authenticity of the Album Covers in the Artefact Submission ...... 52
Synchretic Cutting ...... 55
Peter Bowman Jr.: Exhausting Moment ...... 59
Synchretic Convergence ...... 61
Verbal Signification and Synchresis ...... 62
It’s The Delmer Brothers!: Cindy Doll and L’Efforel ...... 63
Synchretic Collision ...... 66
Synchronism produced via Materialising Indices ...... 67
Pap: The Winnings and Losses of Dr Kyotan ...... 71
Reinforcing a Short-Envelope Collision ...... 73
The Fed-Up Kids: School ...... 74
Synchretic Transmutation ...... 77
Pap: Surfin’ Sluffs ...... 79
Division of Consonance and Dissonance ...... 80
Peter Bowman Jr.: An Arrangement ...... 84
Synchretic Dynamism ...... 88
Dynamic Examples of Mickey-Mousing ...... 89
Mickey Mousing in the Work of Chris Cunningham ...... 90
Thousand Yard Stare ...... 94
ii
Incongruous Dubbing ...... 96
The Fed-Up Kids: The Heel ...... 99
Conclusion ...... 102
UP TO HERE ...... Error! Bookmark not defined.
Reference List ...... 107
iii
List of Figures
Figure 1: The automation of a plugin in Ableton Live...... 37
Figure 2: Re-photographed album covers depicting the recording monikers...... 53
Figure 3: Jump-cutting in the music video for Dirt Off Your Shoulder...... 56
Figure 4: Cutting to the beat in The Hardest Button to Button...... 56
Figure 5: Cutting to black in the introduction of Hey Baby...... 57
Figure 6: Cutting to the beat in Take On Me...... 58
Figure 7: A digital glass breaking effect on videotape footage in Exhausting Moment...... 60
Figure 8: A sequence of images in a Victoria Bitter television commercial...... 62
Figure 9: Verbal signification in a Coast soap commercial...... 62
Figure 10: Synch-points in an All commercial...... 63
Figure 11: Videotaped animation in the music video for Cindy Doll...... 64
Figure 12: A toy projectile being fired in synchronization with a sound effect...... 67
Figure 13: A projectile being deployed with a sound effect...... 68
Figure 14: Collision synch-points with sound effects in a toy commercial...... 68
Figure 15: Glass breaking in the music video for Castles in the Sky...... 69
Figure 16: A droplet hitting water in the Crawling music video...... 69
Figure 17: Zooming in and out to create a synch point in You Need to Calm Down...... 70
Figure 18: Using filters in synchronisation with lyrics to create a synch-point...... 71
Figure 19: An object’s texture and material changing in The Winnings and Losses of Dr Kyotan...... 72
Figure 20: The assignment of objects to instrumentation in The Winnings and Losses of Dr Kyotan. . 72
i
Figure 21: A gap of silence and an accent in the music video for The Middle...... 73
Figure 22: Slapping used as collision in the music video for Betty Davis Eyes...... 74
Figure 23: MIDI and guitar recordings in the Ableton Live session for School...... 75
Figure 24: The layering of video footage in School...... 76
Figure 25: Mobile phone footage re-recorded onto videotape in School...... 76
Figure 26: Animation depicting a hair product's benefits...... 77
Figure 27: The camera being submerged underwater in Butterfly Effect...... 78
Figure 28: Animation to vocal harmonies in Big Time...... 78
Figure 29: The depiction of a shirt being cleaned in a Surf commercial...... 81
Figure 30: Video dissolves in a hair treatment commercial for UpJohn Dermatology...... 81
Figure 31: Morphing used to depict successive vehicle prototypes...... 82
Figure 32: Still frames depicting transmutation in a Strepsils commercial...... 83
Figure 33: Animation depicting the transmutation of fruit into cereal...... 83
Figure 34: High-definition video footage of coloured ink in An Arrangement...... 84
Figure 35: A magnification effect applied to a video recording of cells in I’m Gonna Block You...... 86
Figure 37: Still frames from Musicland depicting the unfurling of a clarinet...... 89
Figure 39: Movements up or down a staircase to a suitably ascending or descending musical phrase...... 92
Figure 40: A seal being placed onto packaging in synchronism with a sound effect in a San Giorgio pasta commercial...... 93
Figure 41: Pasta being squeezed in synchronization with a scalar musical cue...... 93
Figure 42: Light being used to accent the completion of a gesture in synchronisation in a Surf detergent commercial...... 93
ii
Figure 43: Instrumentation assigned to different animated images in Thousand Yard Stare...... 95
Figure 44: A table depicting the planning to assign instrumentation with animation...... 96
Figure 45: Dancers singing the refrain in the Talk Talk video by The Arrows...... 97
Figure 46: A Haribo commercial depicting incongruous dubbing...... 98
Figure 47: A performance by The Beach Boys used in a shreds video...... 99
Figure 48: Headlights in darkness during the final section of The Heel...... 101
iii
A List of Titles Comprising the Artefact Submission
List of Music Videos:
The Fed-Up Kids: The Heel
The Fed-Up Kids: School
It’s the Delmer Brothers!: L’Efforel
It’s the Delmer Brothers!: Cindy Doll
Peter Bowman Jr: Exhausting Moment
Peter Bowman Jr: An Arrangement
Pap: Surfin’ Sluffs
Omitted Compilation:
Pap: I’m Gonna block you
Pap: Grey Day
Pap: Taking over
Pap: Winnings and losses of Dr. Kyotan
Gary War: Thousand Yard Stare
iv
Album and Track List of the Musical Submission:
The Fed-Up Kids – Campus Slump OST (1981) [19:55]
1. School [02:13] 2. Interstate [02:04] 3. Past Curfew [02:38] 4. The Heel [01:45] 5. Before You Take Your Leave [01:59] 6. I Like Toys [01:42] 7. Dread [01:01] 8. I Make a Bet [02:42] 9. Pierre Alviset Recess [01:36] 10. Night to Day [02:10]
Pap – Self-Titled (2019) [32:19]
1. Give it a Close [05:25] 2. Huh, Well [04:47] 3. A Strange Complaint [02:41] 4. Car Park Weep [01:36] 5. The Split Bell [04:49] 6. That’s a Hot Wreck [01:36] 7. Gray Day [02:10] 8. Personal Public [03:32] 9. Taking Over [01:18] 10. Prawn Water [02:02] 11. The Bread and The Fresh [02:19]
Peter Bowman Jr – City Nights (1984) [32:06]
1. Mighty Man [02:42] 2. Nothing in Common [02:37] 3. Exorcism by the Ocean [03:42]
v
4. Dressed as an Insect [02:33] 5. Close to Eleven [03:54] 6. Heaven Wants More [02:26] 7. Research [02:47] 8. Tunnel of Love [03:29] 9. Leaving Hell [01:16] 10. I Didn’t Mean To Have Fun [02:32] 11. Dreams [02:46] 12. Raindrops [01:17]
It’s The Delmer Brothers! – Commercials 1975-1981 [02:21]
1. L’Efforel Advertisement [00:33] 2. Rainbow Shoes Advertisement [00:21] 3. Seed Brew Advertisement [00:29] 4. S.L.P. Figurines Advertisement [00:29] 5. Cindy Doll Advertisement [00:27]
Len and the Dippers – Self-Titled E.P. (1983) [08:22]
1. Dipper’s Dawn [01:51] 2. Rebel Dipper [01:45] 3. Paul’s Dip [01:13] 4. A Creeper’s Dip [01:38] 5. Dipper’s Dream [01:53]
Boats – In-Side [14:34]
1. Speedboats [01:35] 2. Sailboats [01:19] 3. Seasick [01:13] 4. Houseboats [01:07] 5. Paddleboats [02:08] 6. Garbage Scow [01:57] 7. Bell Buoy [00:49]
vi
8. Fog Horn [01:47] 9. Bell Buoy 2 [01:07] 10. Skeleton Crew [00:45] 11. Late in the Day [00:42]
Wurselen Embargoed – Small Hours (1989) [14:35]
1. Small Hours [02:59] 2. Port Security [01:56] 3. Followed by Shadows [02:09] 4. Exploring the Inn [02:28] 5. Making Bread [02:18] 6. Hokfitch Estuary [02:42]
vii
An Overview of the Project
In the widely referenced Audio-Vision, Chion invokes the term synchresis, which he defines as “the spontaneous and irresistible weld produced between a particular auditory phenomenon and visual phenomenon when they occur at the same time” or “the forging of an immediate and necessary relationship between something one sees and something one hears” (Chion, 1994, p. 5 & 75). With a focus primarily on the relationship between moving images and sound in screen media, Chion presents how synchresis is gesturally invoked to exposit narrative information or style, ranging from the bombastic and alarming, to covertly forged and subtle alignments of “sound/image synchronism” (Chion, 1994, p. 5). This project is founded on the lexicon of ideas around audio-visual synchronism presented in Audio-Vision, by using Chion’s terminology to discuss the form of a synch-point, or a discrete instance of synchresis.
As well as being a leader of the forefront of synchronisation, the origins of some of Chion’s use of terminology is attributed to Pierre Schaeffer (Schaeffer, 1966), Chion’s mentor and a musical theorist and composer. While I have chosen Chion as the main theorist to who I return throughout this text, additional ideas and theorists who have written about the conditions of synchronisation by Kathrin Fahlenbrach (Fahlenbrach, 2005), Paul Taberham (Taberham, 2018) and Michael Betancourt (Betancourt, 2007). It is important to note that while there may not be equal distribution of my discussion of a particular theorist’s work, there are several contributing writers to the terminology and discussion throughout.
The aim of the project is to present a taxonomy to classify audio-visual synchronization, which refers to Chion’s discussion from Audio-Vision to break synchresis into types, described herein variously as a synch-point, short and long-envelopes, and synchretic parameters. With a focus on production, the accompanying practice-led submission of music and music videos are supported by an examination of audio-visual synchronism in commercial advertising and music video. The audio-visual submission is presented alongside the written component, which explicates the journey of the research, and proposes a taxonomy of audio-visual synchronism. This taxonomy was developed from the observation of synch-points in music video and commercial advertising, instances of which are the basis for experiments in the various works of the adjoining artefact submission.
The intrigue of this project is an attempt to identify and experiment with ways audio-visual synchronisation can occur, examples of which are encountered in music video and commercial advertising and articulated in production in the songs and music videos which comprise the artefact submission. The artefact submission demonstrates distinct approaches to the acoustic or visual mode 8
leading the creation of a synch-point in screen production. The experiments articulate prescience of either the audio and video in approaches to creating synchronisation: as music in some works was created before the visuals, while in other experiments the visuals were created and music was created afterwards. As such either the video or audio are created first and become the guiding mode through which the following mode is translated. This I would suggest is the unique aspect of this project, which investigates instances of synchresis in the practice and discusses them, then articulates my own uses of these expressions.
The focus of this project is the production of screen-based synchronization in a film or video sequence. The practice-led approach which has allowed for experimentation with a range of audio and video recording media, to produce a varied groundwork to creating synch-points with a range of expressive noise that is indicative of previous eras of cultural production. To distinguish between these approaches, the artefact component of this project is presented as an archive of lost albums and music-videos, which is hosted online on the music streaming platform Bandcamp.
As a submission of practice-led research, the project herein is comprised of a written component, alongside an accompanying submission of music and music videos. The artefact is presented as an archive of albums and music videos from a fictional record label: Hot Off the Press Records. These fictitious collaborations by non-existent musicians, music-video directors, photographers and graphic artists, each articulate a different approach to the moments of synchronisation in a video sequence. To identify different kinds of synchronisation, four synchretic operations, examples of which are discussed in the sites of music video and commercial advertising.
The submission of audio-visual experiments are framed as the creative output of musicians and their adjoining music videos and album covers, which have been re-released as an archive by a contemporary record label: Hot Off the Press Records™. These various monikers embody different approaches to synchresis and the media used in the production process, which are presented as the output of individual artists: Peter Bowman Jr, Len and the Dippers, Wurselen Embargoed, It’s The Delmer Brothers, Boats, Len and the Dippers, the Fed-Up Kids, and Pap. For each moniker aside from the two collaborative projects Pap and Len and the Dippers, I have recorded and sequenced the music, music-videos and accompanying album covers which comprise the artefact submission. The total length of the musical submission of the artefact includes four extended play albums and two long play albums, which together amount to approximately 120 minutes.
9
A Discussion of the Screen-Production Research Methodology
Recently established areas of research, such as creative writing, dance, theatre and performance studies, film television and media studies are often marginalised, with an artefact submission remaining contentious as a legitimate and viable research outcome when compared to a written submission (Nelson, 2013, p. 24 and 25). Practice-based Research (PbR) is typically accompanied by a written component alongside an artefact as “substantial evidence” of the enquiry, which in this project is a submission of music and music videos to illustrate audio-visual synchronisation in production (Nelson, 2013, p. 8 and 9).
Practice-led research is inherently messy as the knowledge that emerges from the creative process is “immersive and nervous, more implicit than explicit”, which is reflected in this project by the varied approaches to fusing a synch-point, the array of media and recording technologies used to produce them and a range of recording monikers to explore different musical and visual styles (Gibson, 2010, p. 6). The connection to a written component acknowledges the practice component as a “messy initiator” and provides context for the research findings, in which the acquisition of answers and definite outcomes are restless and by nature of discovery-based are complex and unstable (Combrink & Marley, 2009, p. 191; Gibson, 2010, p. 6).
The creation of audio-visual synchronization is central to the project, in which the artefact was developed throughout years of incremental learning, of arduously developing skillsets and imbibing audio and visual media production techniques. The music and music videos were developed through training and gestation, during which ideas were incrementally “wrought into realisation”, through “playful experimentation” with the materials of the process and the ideas that emerged from their execution (Nelson, 2013, p. 28; Rahn, 2008, p. 305). As the production of audio-visual media required extended periods of training, reflecting the idea that within the industry “learning by doing” is widely accepted as the most acceptable means of developing skill (Hanney, 2013, p. 46). It has been argued this approach allows the researcher to embrace the communication of knowledge through the artistic process, by stepping “into the mystery”, between the complexities of “action and reaction” until a shift has been observed (Gibson, 2010, p. 5).
The composition of music and the recording and editing of music videos is central to evidencing the enquiry itself, in which “knowing-doing” is imbricated within the practice (Nelson, 2013, p. 9): as the central focus of this project is experimenting with articulating synchronisation myself, I have attended the intricacies of generating the production of the artefact from start to finish. These audio-visual experiments articulate distinct types of audio-visual fusion, according to the classifications described in the taxonomy as collision, cutting, dynamic and transmutation. These synchretic operations are 10
distinguished by their duration and the means of their emergence: whether they are accented or sustained; a fusion produced via the collision of object; a correspondence between visual and acoustic texture or the materialising indices; the juxtaposition of shots during the post-production stage via cutting; the directional movement of camera movement or motion in the mise-en-scene; or a gesture of morphing initiated from within an object.
Synchresis emerges via a fusion of audio-visual information across a timeline, and is produced via the media used at each stage of its production; it is unique to this project that the practice-led exploration of audio-visual synchronism could not be fully articulated solely by linguistic explication, or by way of a traditional thesis using “words alone” (Nelson, 2013, p. 9). Given this, by a submission of theory and practice “imbricated within the other” (Nelson, 2013, p. 29), the value of the project as an examination of audio-visual synchronization in screen-media is to:
1. Suggest a taxonomy of the form of a synch-point, comprising short and long-envelopes and parameters;
2. Using the production of various audio-visual experiments: mobilise the taxonomy in a practice-led approach submission, rather than solely with an exegetical component; and
3. Elucidate the fraught aspects of the research project with an examination of audio-visual synchronization in the submission of artefacts alongside an examination of synchresis in music video and commercial advertising.
Ross Gibson states the creation of an artefact is meaningful as art, and research can effectively make knowledge, when in their partnership they dispel “ignorance or befuddlement” (Gibson, 2010). The appropriate object of the knowledge through creative practice research is a focus on outcomes that supply insight to the “generative performance” of a work’s production (Bell, 2006, p. 98). Gibson describes the process is less a schematised blueprint, but a set of narratives which explicate the changes and “encourage speculations” regarding the complexities of the system central to the research enquiry’s dynamics (Gibson, 2010, p. 9).
In this project, the system that is under scrutiny is the taxonomy to classify synch-points that form the basis for the production of artefacts in addition to the discussion of examples in commercial advertising and music video. The music and music videos correspond with the accompanying written component herein, which are submitted in combination to document the exploration and explain the research process (Winter, 2010, p. 3). The process of reflection in the discussion of recording and sequencing the artefacts is intended to address the necessity for a written submission, to evidence the research
11
with a document which is a repository for binding the information in the process, entailing “taking stock, evaluating and off-loading” the research findings (Combrink & Marley, 2009, p. 190 and 191).
The Screen-Production Framework of the Project
In a foreword to a volume on screen production research, Gibson takes the position that the goal of producing work as filmmakers in the academy is “pre-eminently” knowledge production rather than film production, arguing that film production is a valid research venture through which “the academy employs us” to generate and communicate new knowledge (Batty & Kerrigan, 2018, p. vi). Gibson outlines steps for a production-led research enquiry, which he notes if followed assiduously, can guarantee a “filmed and edited venture” be regarded as scholarly research (Batty & Kerrigan, 2018, p. ix). To address Gibson’s detailed path to research through film or video production, the following paragraphs respond to the first, second, third, fifth, seventh and eighth of Gibson’s steps, or “arc of connected activities” in relation to this project, to discuss its value to research as well as to outline how the project developed through various iterations. To only address the relevant criteria, I have omitted addressing the points of Gibson’s discussion, such as declaring the research process out-loud, the implicitness of sharing the artefact online, or starting a new research enquiry as the final steps.
1. Identify a hunch
Gibson (2018) states the first step should be to identify the curiosity of the project, which stems from my fascination with synchresis in being pervasive and fundamental to screen-media, though an under- investigated aspect of media theory, in which the focus was solely on the moment of synchronisation itself is taken as a focus of investigation. As synch-points in constellation can audio-visually convey emotion or communicate the details of a message, these gestures of video and film production fascinated me for their communicating in fleeting moments of audio-visual fusion. As a pervasive aspect of screen-media and audio-visual practice, regardless of genre or scale of the production, synchresis is fundamental to narrative modes of storytelling in cinema and television production, early autographic and cel animation, 3D animation, commercial advertising, while also being key to the histories of the avant-garde and UGC (user-generated content) on YouTube.
12
Audio-visual synchronism was key to my honours thesis submission, the fusion of which I perceived exceeded how I had previously encountered it as a rare occurrence in a sequence, than as the main interest of a video work (p. ix). This initial fascination came via Chris Cunningham’s music videos and commercials, in which the frequency and rapidity of synchronisation demonstrated the careful detail and revision required to produce inextricability between the video and soundtrack. This interest was not solely for high-speed synchronisation, as a range of music videos in his body of work depicted languid fusions, while maintaining relational changes between the distinct audio and video elements. In contrast to how I had previously focused on narrative considerations, these works explored fleeting and microscopic details of images to sound as the basis for entire works. As a starting point for my interest, these videos and the constant use of synchresis displayed the power of returning to these primal encounters of sound and image, prior to any consideration of narrative.
2. Identify precisely what you want to know or need to know – give language to this intrigue – so you can begin to satisfy the urge by composing and following a communicable plan
During the nascent stages of the project, the plan for the artefact submission and written component underwent restructuring due to navigating the related research areas. Given the relevance of audio- visual synchronism to a breadth of pursuits, to compose a fully communicable plan for the research, I surveyed literature in the areas of cross-modality, the phenomenology of various screen-media, semiotics and various branches of art and media aesthetics. While keeping Chion’s (1994) Audio- Vision a central text to the project, the broadly relevant areas emerged from related areas, such as: the early histories of filmmaking and animation (Beller, 2011; Carroll, 2003) the leap to sound film (Altman, 2005; Zavrl, 2015), automated synchronisation and video disc jockeys (Faulkner, 2006; Majoe, Kulka, & Schacher, 2009), avant-garde filmmaking and visual music (Jones & Nevile, 2005), sound design in animation (Beauchamp, 2013) and debates surrounding the phenomenology and epistemology of the moving image (Branigan, 1989; Morgan, 2006; Wood, 2001).
While they are beyond the scope of this thesis, of which I necessarily relegated to examples in music video and commercial advertising, audio-visual synchronism also connected to areas of research examining human reception and interaction while not being explicitly referred to using the term synchresis, such as research in synaesthesia (Carlson & Gorman, 1990; Cytowic, 1993; Marks, 1982), perceptual science (Melara, 1989; Walker et al., 2010), cognitive science (Andersen, Tiippana, & Sams, 2004; Currie, 1995) and the development of cross-modal interactive technologies (Jones & Nevile, 2005; Nanayakkara, Wyse, Ong, & Taylor, 2013).
13
As a result of exploring the literature related to audio-visual synchronization in research surrounding cross-modality and perceptual correspondence, the project during the burgeoning stages was envisioned as a series of trial led-procedures. During these early developments, the planning reflected the research literature in cross-modal studies relating to synchronisation in the area of sense perception rather than screen-production. Earlier iterations of the research plan were intended to be undertaken as a series of cross-modal trials to observe the sensory interaction of audio-visual synchronisation, with the intention of obtaining qualitative data.
Given the related scientific disciplines are not my area of study, a cross-modal trial-led procedure would have situated the research further away from the area of screen-media and into the area of “neurogrammatological” research (Halliday, 2013, pp. 90-91). Following my initial failed experiments to form a coherent and unique approach to a trial-led procedure, I focused solely on my own ability to form synchresis in audio-visual production, by articulating various kinds of synchronization in the artefact submission, as a project which articulates examples in the production of “researcher-created data” (Kerrigan, Leahy, & Cohen, 2016, p. 22).
3. Check (by means of in-depth investigation of existing knowledge) whether this identified intrigue is something the scholarly community wants or needs to know about. If yes, proceed; if no, revise.
To research disciplines surrounding the synchronisation of sound and image in screen-media, the need for this project is in the investigation articulated in the production work, in which the types of synchronisation and their differences cannot be experimented with or understood solely from this accompanying text. To present categorical separation between instances of synchresis, the suggested taxonomy of types of synchronisation has been formed from identifying instances of synchresis in music video and commercial advertising. From identifying uses of synchresis in music video and advertising, the artefact articulates experiments with the aesthetic conventions of industrial uses of synchresis in screen-production, while they are uniquely explore the freedom of being un- commissioned works, that can be changed and are mutable as they are being made.
In the present study, part of the importance of is to explore synchresis of which the music and video are developed interchangeably, as the simultaneous development of music and music video is not often afforded by or the focus of commissioned collaborative works. The artefact explores the prescience of both audio and video: distinct examples in which audio is created first and leads the expressions of the moving images and those in which the video leads the creation of sound. The
14
creation of the music is integral, as the exploration of synch-points fused using either audio or video as the prescient mode require the ability to create and manipulate both. The production of music also would allow the application of noise and distortion at varying stages of the work, by which the materiality of synchresis is articulated.
Materiality and medium-specific noise are used descriptively, as an aspect of the contribution of this project is to articulate the leading mode increasing in noise or distortion, as the following mode increases in noise or distortion to synchronise. The range of recording technology used foreground the differences between recording media, which through the development of the project became central to articulating noise and distortion in a synch-point. The medium-specific noise section addresses the interplay of changes in noise and distortion to produce synchronisation, using a range of recording formats, bitrates, dynamic ranges, analogue and digital tape stock, film and ISO type, cassette type, and the processes of degrading the footage using hardware and software filters. The medium-specific noise and distortion in the project can be regarded as the materiality of the synchronisation, as the noise and distortion and recordings produced via capturing errors are key to the use of particular recording media.
To illustrate this complexity, visual grain might be added by way of the render settings in a 3D object, or alternatively be added due to a high ISO, alternatively this may correspond to a software generated grain particles, which might also be added in later stages of post-production using an alpha channel which has 35mm stock grain. In other words, the same effect of video grain which may be indiscernible in its difference even to a trained eye, may be achieved through a variety of processes. In discussion, the terms distortion and noise would be used in reference to the combination of these process, as it is beyond the scope of this project to articulate the detail of each contributing aesthetic element.
The investigation of materiality is related to attending the moment of a synch-point’s creation, which would not have been possible if a participant were required to learn how to perform and edit a sequence themselves. As I have focused on synchronisation in screen production for nearly a decade prior to the undertaking of this project, I have undertaken the process of combining sounds and images requires a knowledge basic video editing skills, as well as the willingness to undergo the necessary trial and error to produce audio-visual synchronisation. This would be unattainable if the participant were required to attend to the creation of both the audio and video and to focus on producing certain relationships, such as restricting synchronization to occur between selected audio and visual elements. As audio-visual synchronization in video production would require long periods of trial and error and of reflection during the process, to achieve the study at the level of production myself would allow for control over recording and sequencing both the audio and music, with a breadth of media and
15
expressive noise imparted onto the recordings by ongoing experimentation with analogue and digital recording methods.
4. Focus the quest with one or more questions (so that the questions will motivate and constrain the scope of the quest all the way through the rest of the research process) 5. Devise and declare the method
With a basis in the terminology, suggested exercises and discussion Chion (1994) presents in Audio- Vision, the research question this project asks is, “What are the constituent elements that comprise the form of a synch-point, and how can they be identified and used in screen-production?”. Having abandoned a trial-led procedure, the focus of the project shifted to an exploration of synchresis in production rather than reception, by developing a taxonomy to articulate an instance of audio-visual synchronism in its constituent elements. The interplay between the written component and the production of synchresis in the artefact submission are informed by Chion’s (1994) discussion, terminology and exercises for producing audio-visual synchronism.
I suggest each synch-point or single-instance of audio-visual synchronization is comprised of audio and video parameters which adhere to a synchretic envelope. Of the broad categories of long and short envelopes, I have identified four kinds of synchretic operations: collision, cutting, transmutation and dynamism, which are identified and discussed in the sites of commercial advertising and music video, as well as forming the bases for distinct audio-visual experiments in the submission of artefacts. With the aim to form synch-points, the range of musical output of the different monikers is intended to provide a groundwork to explore the relationships of “rhythm, intensity, and primary gestalt patterns” (Fahlenbrach, 2005, p. 7).
While some articulate musical experiments in short-envelope staccato notes and accented rhythm tracks, others exemplify long-envelope notes that are illustrative of sustained notes and languid instrumentation. Certain musical recordings are written with the intention of synchronising to a particular kind of video, for instance to imitate the texture of an object with the quality of the sound, or in Chion’s (1994) terminology the materialising indices of a sound. This was to illustrate the texture or material aspects of a sound which change or morph throughout their playback, to illustrate a visually synchronised change via examples of transmutation, such as The Winnings and Losses of Dr Kyotan.
In Exhausting Moment the acousmatised or unseen sources of sound are synchronised with the song, to imply the sound off-screen is the leading mode for providing a synch-point. In Thousand Yard Stare 16
each instrument is shown simultaneously, while An Arrangement by Peter Bowman Jr explores transmutation of a synch-point between sustained notes and close-ups of moving dye. The music- videos of The Fed-Up Kids moniker use live-action footage to short accented notes and a single melody, while the experiments of the It’s The Delmer Brothers! moniker explore the synchronisation that is typical of vintage advertising.
6. Generate the knowledge, using the method. This is an iterative, creative and cumulative and sometimes recursive and revisionary process.
The generation of knowledge in this project is produced via the submission of music and music videos, of which the individual procedure is discussed in the second half of this text. In focusing on either the presciently created audio or video, the process was, “iterative, creative and cumulative and sometimes recursive and revisionary process” (Gibson, 2018, p. x). While some of the experiments in synchronization were achieved quickly or accidentally, other works in the artefact were built on through processes of collection and added to incrementally over the duration of several years. In illustration of the “the immersed, messy routines of creativity oscillating with the distanced analytics of reflective critique and theorisation”, the project and its findings have not been as clearly bounded as I would prefer or had intended.
This section is followed by an overview of audio-visual synchronization to discuss the voluble, wide- ranging application of synchresis in screen-media. Informed by a discussion of Chion’s terminology in Audio-Vision (1994), the text then describes the taxonomy I have suggested for the constituent elements of synchretic envelopes, parameters and operations. This is followed by an introduction of the two key sites of music video and commercial advertising, in which key examples of collision, cutting, dynamism and transmutation are identified.
To acknowledge noise and distortion as a contributing aspect to synchresis in the submission of music videos and music, the analogue and digital recording media are discussed in the production of the artefacts. This discussion is followed by the identification of instances of collision, cutting, dynamic and transmutation in the sites of commercial advertising and music video, which are interwoven with descriptions of the artefact submission. In summary, the structure of the thesis begins with an overview of synchronization, the taxonomy for various different kinds of synch-point, the acknowledgement of the various areas of the project, and a discussion of how synchresis is produced in the artefact submission.
17
As a mix of artefact and exegesis, the project provides an important opportunity to advance the understanding of the growing area of research surrounding audio-visual synchronisation. The study aims through a unique exploratory method to develop both the audio and video simultaneously in production. The importance of the simultaneous development of the audio and visual in the artefact, is unique for a study of synchronisation, as it allows for the articulation of synchresis by using presciently created video, or audio, or by developing both simultaneously. In this way, the methods of the production articulate the differences between types of production and the approaches to process which impact the creation of a synch-point.
18
A Brief Overview of Audio-Visual Synchronization
Audio-visual synchronism is innate to creative pursuits and multimedia arts-based practice with the ambition to link moving images and sound, such as film and television production, commercial advertising, experimental film, video art, UGC (user generated content) online, multimedia art- installation and live music visualisation (Coulter, 2010, p. 26). The use of the term synchresis in this project is consistent with Chion’s (1994) discussion in Audio-vision, which is used in reference to audio-visual synchronism in the domain of screen-production, rather than focusing on areas to which audio-visual synchronism is relevant, such as automated synchronisation in live-performance or cross- modal and perceptual phenomena research. While it is beyond the scope of this text to summarise the history of synch-sound and to convey its impact, this section provides a broad overview of audio-visual synchronisation.
The earliest experiments of combining visual and audible stimuli are situated in the pre-cinematic arts, having been a desire for artists and inventors for over one hundred and fifty years (Hausken, 2013a, p. 218). This is evidenced in the origins of artistic endeavours that have sought to combine the audio- visual, that fuse modal qualities of sound and musicality, such as “rhythm, pitch, duration, and tone colour” with the formal elements of the visual (Diederichsen, 2011, p. 121). In the nineteenth century, attempts to visualize sounds were undertaken by Josef Matthias Hauer by using Goethe’s theory of colours to assign tonal values and intervals with a colour wheel (Corbella, Windisch, Gaudreault, Barnier, & Sirois-Trahan, 2013, p. 67; Hausken, 2013a, p. 223; C. O'Brien, 2005, p. 3).
Around 1665, Isaac Newton observed relationships between optics and musical tones by creating an analogy between the seven colours of the spectrum and the seven notes of the diatonic scale (Pesic, 2006, p. 294). In 1787, the German physicist and inventor Ernst Florens Friedrich Chladni studied the nodal lines of vibration to develop musical instruments, such as the clavicylinder and the euphonium (Engberg-Pedersen & Simons, 2018; Jueneman, 1998, p. 11). These early experiments would graphically transcribe sound vibrations, that were produced by scattering sand onto a metal plate and vibrating it with a bow, causing the sand to collect in a symmetrical figure representative of the sound, what Chladni called Klangfiguren or “sound figures” ("Chladni, Ernst Florens Friedrich," 2002; Engberg-Pedersen & Simons, 2018, p. 329; Ribas, 2016).
Popular entertainment forms in the theatrical arts, such as vaudeville, opera, and the broadcast media of radio and newsreels assisted in shaping the Chronophone, Phonofilm, and Movietone sound synchronous sound technologies (Zavrl, 2015, p. 35). Leon Gaumont’s Chronophone was developed in 1902 and intended for vaudeville shows, by linking via cables two phonographs and a film projector. 19
In achieving synchronisation in a live-context, lip-synching was performed to music of a phonograph disc, which accompanied the moving image during screenings to give the impression both were recorded at the same time within the same space (Zavrl, 2015, p. 35). These screenings have a likeness to publicly screened music videos, for their depiction of a performance of lip-synching to the recorded playback of a song.
The initial endeavours to combine sound with moving images from the 1890s to the 1920s have a history of treading between success and failure, as research has evidenced experiments prior to Edison’s Phonograph in 1877 or a variation thereof, with many attempts being aborted prior to their realisation (Geduld, 1975, p. 3). The introduction of speech was initially viewed by critics and filmmakers alike as a disadvantage to a fundamentally visual medium (Jacobs, 2014, p. 1; Spadoni, 2007, p. 8), while musical accompaniment initially fulfilled the utilitarian function of masking projector noise (Warner, 2017, p. 168). Following intensified efforts to develop sound synchronism during the transition from silent to sound film (1926-1935), the end of the silent period marked by talking pictures signalled the paradigm-shift towards cinematic realism (Carroll, 2003, p. 12; Corbella et al., 2013, p. 61/62). The ability to record and synchronise sound transformed the phenomenology of the moving image, as electronic amplification, optical soundtracks and tight synchronization added new possibilities to film language (C. O'Brien, 2005, p. 3).
The sole remaining example of audio-visual synchronization from 1894 at Edison’s laboratories is a musical performance, which depicts the head of the motion picture division, W. K. L. Dickson, performing the 1877 composition The Chimes of Normandy by Robert Planquette on violin (Kalinak, 2010, pp. 32-33). As the earliest cited example of musical accompaniment fused to moving images, the test for the Kinetophone depicts two male employees dancing as Dickson plays the violin, the wax cylinder soundtrack of which had been lost for almost a century until its repair and synchronisation with the moving images in 1998 (Kalinak, 2010, pp. 32-33).
Early examples of combining film with sound performed on a stage vary between the country of production, which is evident in cinematic history prior to invention of the optical soundtrack. For instance, the Japanese silent cinema had recourse to the Benshi, a man seated off-stage who would vocally describe the film (Fujiki, 2006, p. 71; Gerow, 2010, p. 133; Warner, 2017, p. 168). As opinion polls from the period suggest, Benshi were often more popular than actors in the films, although critics cited the incompatibility of stage performers with the screen action, encouraging Benshi to abstain from voice impersonations or kowairo and instead to provide the names of the characters, locations and aspects of the historical and cultural background (Bernardi, 2001, p. 178; Fujiki, 2006, p. 71). It has been argued that in the act of assuming the role of narrating the film, the ocularcentrism of the
20
medium is defeated with the addition of a Benshi, as the addition of a spoken and performed accompaniment “purges the text of narrative itself” when the moving image cannot articulate the narrative solely by way of the visual (Gerow, 2010, p. 134).
At cinema’s arrival in the 1890s, the use of performed sounds accompanying live demonstrations was a standard part of public lectures, which were often accompanied by music (Neumeyer, 2014, p. 18). Bottomore notes the use of synchronised sound effects used since the Middle Ages in theatrical productions, at which the sound of animals, rain and thunder were produced by “thunder runs” and “thunder sheets” (Bottomore, 1999, p. 485). Similarly, the magic lantern: a pre-cursor technology to cinematic apparatus rose to prominence throughout the nineteenth century, being used for religious and pedagogical purposes during lectures, in schools and churches (Bak, 2015, p. 112; Gunning, 1995a; Schaefer, 2017). These lantern projections also display evidence of early sound effects of thunder, wind and “eerie voices” (Bottomore, 1999, p. 485).
During the silent era, sound belonged to the performance rather than the film itself, which functioned to underscore the narrative by extending musical cues across shots to bind them in a sequence (Neumeyer, 2014, p. 20). Music was not indifferent to the visuals, as it was temporally fit with the moving images and did less to mimic the visuals rather than compliment the “pertinent aspects” of a narrative sequence (Neumeyer, 2014, p. 20). The musical directors judgement was paramount to the accuracy of the synchronisation, as well as the accuracy of the information on the cue sheet, and that the success of musical synchronisation was due to the musical director’s accuracy of judgement rather than careful timing (Beynon, 1921, p. 102; Neumeyer, 2014, p. 20).
Synchronisation in the early stages of moving image history included in a cue sheet, which would give important information about the accompanying scenes. The widespread use of the term throughout the late teens and twenties, refers to the accompaniment instructions for musicians, to provide the points at which an instrument should enter the arrangement (Altman, 2001, p. 234). The sheets would also indicate the suggested music by an arranger employed by the motion picture studio, to account for each scene or major action in the film, the approximate lengths of scenes and when to use sound effects (Leonard, 2016, p. 260 and 261).
With a likeness to cue sheets, special scores provide access into precise synchronization prior to synchronized sound, which are distinguished from cue-sheets by their “ready-to-go” nature (Brown, 2013, p. 583). These as Leonard notes were original soundtracks designed to be paired with a particular film, which were often created for historical dramas and operas (Leonard, 2016, p. 259). In Edith Lang and George West’s Musical Accompaniment of Moving Pictures (Boston: Boston Music Co.; New York: G. Schirmer, 1920), the authors explain the use of theatre organ, “musical 21
characterisation, improvisation, and transposition”, along with instructions for how to use the theatre organ to create effects (Leonard, 2016, p. 262). Lang and West suggest particular keys, which could be invoked to reflect warmth, brilliancy or sorrow, for instance: while E-minor was suggested for the expansive images of the sky or ocean, F was appropriate for religious or meditative subject matter (Boller, 1985, para 20).
Certain scores were available in different iterations, as for D. W. Griffith’s Intolerance (Griffith, 1916), depending on the location of the screening, version were made for a full orchestra, for pianos and organs for smaller theatres, and scores marketed to viewers who wanted to play the score at home (Leonard, 2016, p. 260). May Meskimen Mills on the other hand linked types of music to suit particular actors or actresses, recommending for instance that Douglas Fairbanks should be accompanied by “humorous, snappy, fast and furious” accompaniment (Boller, 1985, para 20). As the first 30 years of public screenings did not have original music but were synchronized to a gramophone in the exhibition space, there were difficulties of rejoining a film with its original soundtrack, (Brown, 2013, p. 583). Performance was intrinsic to the early sound-synch experiments, which to maintain synchronization had a standard and consistency, using cue sheets, special scores and the early recordings of soundtracks.
In addition to cue sheets, Julie Brown notes that special scores provide access into precise synchronization prior to synchronized sound, which are distinguished from cue-sheets by their “ready- to-go” nature (Brown, 2013, p. 583). These as Leonard notes were original soundtracks designed to be paired with a particular film, which were often created for historical dramas and operas (Leonard, 2016, p. 259). A compiled special score “contained instructions for performers to improvise as they saw fit on given musical themes, although as the many articles listed in the bibliography can attest, the appropriateness of improvisation was a hotly contested issue in film accompaniment” (Leonard, 2016, p. 260).
In 1928, Rudolph Arnheim suggested the audience had no preference for whether the sounds were produced by a machine or performed by human beings (Arnheim, Benthien, Newbury, & Diederichs, 1997). As his opinion changed over time, in coming to recognize the viability of non-synchronous sound, Arnheim considered sounds that were not anchored to the moving images to be inherently ambiguous (Neumeyer, 2014, p. 26). Arnheim contended the image served as the anchor for sound, and that off-screen sound, should while setting up an “expectation for synchronization”. The talking film for Arnheim is “irreducibly mimetic”, due to the way synchronisation foregrounds whatever is synchronized, while vectorizing time, and “so unbalanced all attempts at nonsynchronous sound” (Neumeyer, 2014, p. 26). Neumeyer suggests Arnheim regarded the “literal realism” imparted by the
22
mimetic quality of sound film as being limited by the “inherent ambiguity of any sounds not firmly anchored in the image” (Neumeyer, 2014, p. 26).
Bottomore notes that as early as 1896, it was regarded as disturbing for a lack of sound to accompany a familiar action, as in for instance images of a blacksmith which are not accompanied by hammering, the sound of the steam as the hammer was plunged into water (Bottomore, 1999, p. 487). The lack of accompanying sounds in a Corbett-Fitzsimmons fight film screened in 1897 in Sydney was criticized as being an alienating, that it could be improved if the management were hit a hung piece of beef with each strike, noting it could add make the illusion more realistic “provided the beef was smote at the right moment” (Bottomore, 1999, p. 487).
To classify the types of musical relationships between the vision and audio in screen-production, Beller (2011) suggests there are two historical and ideological factions between moving images and film music, the first of which is music which is complimentary and oriented to the action, whereby the musicality accentuates the images and the plot in the aim of unifying the content (p. 104). The other is a relationship that is independent of the moving image, which in its expressive opposition and independence finds a dissonance to the image and thereby enacts “a commentary on and counterpoint to the image” (Beller, 2011, p. 104). While the affirmative or Illusionistic faction reflects the consumption-orientations of the Hollywood studio system, the alienation faction invoking the Russian concept of montage demands contrast and conflict to the moving images (Beller, 2011, p. 104).
Coulter assigns the terms congruent for audio-visual relationships or those that go together as concomitant, “accompanying, concurrent, attendant”, suggesting that synchresis occurs as the pairing of audio and visual information exhibit “concomitant relationships” or “assume primacy” by way of their “homogeneity” over those that do not (Coulter, 2010, p. 27). Pauli suggests paraphrasing, polarisation and counterpoint as three categories of music and image: paraphrasing describes music as being “additive” or “congruent” with the content in the image, “polarisation” (the music disambiguates the scene) and “counterpoint” conveys an ironic comment on the image (Collins, 2013, p. 26).
In the submission of artefacts, the music video for Exhausting Moment by Peter Bowman Jr. explores synch-points to different acousmatised audio sources: a term formulated by Schaffer (1966) in the Traité des objets musicaux and discussed by Chion (1994) in Audio-Vision to refer to the source of an audible sound not being visible (Kane, 2014, p. 17). Schaeffer discusses the definition of acousmatic from the Larousse, stating “a noise that is heard without the causes from which it comes being seen”, which for a listener presents an exercise in which listening itself is the phenomenon in question
23
(Schaeffer, 1966, p. 64). The exercise is to distinguish between a sound’s production and transmission methods, and as such emphasize “the perceptual reality of sound” (Schaeffer, 1966, p. 64).
While it is simple enough to observe the difference between a violin or a creaking door, as by identifying it as such the listener is “alluding to the sound produced by the violin, the creak of the door” (Schaeffer, 1966, p. 67). The distinction that is intended is more radical than identifying the causal reference of the sound, as suggested by the example of being a recording of a sound without an identifiable origin. What is heard in such circumstances of an identifiable origin, is what Schaeffer calls a sound object: “independent of any causal reference covered by the terms sound body, sound source, or instrument” (Schaeffer, 1966, p. 67). As a means of this is to focus on the “object” itself, rather than “as a vehicle for something else”, Chion also uses the term reduced listening from Schaeffer, in reference to listening to a sound while focusing on the musical traits, regardless of its cause and any potential meaning (Chion, 1994, p. 29).
Musical synchronization of action to sound in cinema and theatre is challenged by the history of animation, in which resounding examples of visualizing sound effects and music are found, which similarly to a traditional film score are used to draw audience into story, establish setting and emotion and vitalize the moving images (Goldmark, 2005, p. 7). Given the importance of this area to a project concerned with audio-visual synchresis and to the animation practices that inform the production of the artefacts in this submission, the following section is an overview of synchronization in animation.
24
Animation and Synchronous Sound
With its basis in the shifting rhythms, motion and speed of form and colour, Wells suggests if music were visualized it may have a likeness to experimental animation: characterizing the early output of Warner Brothers and Fleischer Brothers shorts as early music videos, that draw narrative imperatives from the music (Wells, 1998, p. 46). Taberham suggests four traditions of synchronisation in cartoons, which each progressively become less reliant on being led by a musical structure and adhere less to a musical sound effects. During the synchretic period and the most relevant mode to this project, is what Taberham suggests the term synchretic cartoons, in which the onscreen movements conform by design conform with music which been developed from the start of the creative process (Taberham, 2018, p. 132).
In the synchretic tradition of the 1930s which were led by the Disney studio, there is a prominence on musical expression, during which productions use the soundtrack to respond to musical rhythm and conventions, rather than “stretching” to adhere with the images or story (Taberham, 2018, p. 132). Taberham states that core to the style is a focus on music, in which dialogue and sound effects are integrated into the rhythm of the film (Taberham, 2018, p. 132). Central to this mode is a fixed tempo over the course of the short, as the tempo prescribes how long the walk of a character, the amount of steps and the speed of their movement to the rhythm of the music (Taberham, 2018, p. 132).
In being intertwined with the burgeoning stages of cinema, the placement of sound effects during this era are often musical in nature and performed by studio musicians: “typically slide whistles, cymbal crashes, bulb horns and timpani drums; the same techniques used by pit drummers to produce sound effects during the silent era” (Taberham, 2018, p. 135). Similarly to the use of cue sheets, Furniss describes how at the Fleischer studio a manual was given to employees to indicate the specificities of synchronizing lip-synch to dialogue, and to indicate how the action should develop with relation to the soundtrack (Furniss, 2017, p. 84). Following the assembly of dialogue takes into sequence, the process of synchronisation would take place on paper, from the head animator underlining syllables which were most suitable for accents (Furniss, 2017, p. 84).
The new sound technology afforded a way to distinguish Disney from his competitors, via what he deemed cartoon sounds “extreme novelty” (Barrier, 2003, p. 50). During the synchretic era of the 1930s, the synchronisation to music in Disney’s cartoon’s gave rise to the term mickey mousing, which as Goldmark notes was coined by David O. Selznick in “derisively likening a Max Steiner score to the music of a Mickey Mouse cartoon” (Goldmark, 2005, p. 6). As Furniss describes in 1928 close- synchronisation was seen as a novelty to be utilized, though as the novelty came to wear off the term came to acquire a negative connotation, which has been used to described synchronisation that had 25
been “too tightly matched (Furniss, 2017, p. 91).
With developments in technology facilitating new areas of experimentation, following the synchretic tradition of the 1930s, the zip-crash mode of synchronisation popularized during the 1940s and 50s led by the production of Warner Bros. and MGM became dominant (Taberham, 2018, p. 136). In breaking away from the synchretic tradition, in addition to greater sound fidelity the main difference was dialogue and sound effects did not integrate and respond rhythmically to music (Taberham, 2018, p. 137). Of this era Taberham describes highly stylized voices characterized by “real world” causal recordings of sounds rather than instrumentation, though these avoid resemblance to the physical world from the “flamboyant and incongruous” nature of the combinations, such as tyre screeches as a person halts (Taberham, 2018, p. 136 and 137).
Similar to the use of cue sheets in early cinema, the composer Ingolf Dahl recounts the approach to animating synchronization cues from a detail sheet as a cartoonist on a Tom and Jerry cartoon, via annotations placed on empty staves breaking down the visual action to the musical score. The synchronization of music to a dance routine was recorded in pre-scoring prior to the drawings, to ensure the synchronization of visuals with the music (Dahl, 1974, p. 97). This process would entail the tabulation of moving images according to regular frame units to which the animation was created, to a non-deviating time-unit to represent a beat as number of frames, and thereby indicate the scene’s “smallest rhythmic denominator” (Dahl, 1974, p. 97). To respond without warning would cause problems for the pacing, as “expediency” and a variety of techniques were required to accomplish the required sudden shifts to “accomplish changes of mood, of character, of expression” (Dahl, 1974, p. 98).
In the tradition of Absolute Film (referred to synonymously as Pure Cinema, Integral Cinema, Avant- Garde, and Experimental Film) Visual Music contains early and resoundingly complex examples of fusing images to the musical attributes of “rhythmical form, structure and harmony” (Mollaghan, 2015, p. 1). As a practice, Visual Music refers broadly to the “representation of musical ideas or material through ocular media” (Hyde, 2012, p. 170), resounding examples of which are found in the work of Mclaren (1949), Ruttmann (1921), and Lye (1936). Alternatively, Moritz (1986) offers a definition for Visual Music as “a music for the eye comparable to the effects of sound for the ear” in which visual structures are created to match the intricacy of music.
In the animations of Oskar Fischinger, autographic or hand-based techniques explore the rhythm of musical cues (Castello-Branco, 2010, p. 32). Fischinger’s work demonstrates unique animation techniques, by using a blade synchronised to a camera shutter to create time-lapse sequences depicting the cross-sections of wax being removed (Frederick, 2013, p. 235). These experiments serve 26
to draw out the constituent musical elements of melody, rhythm, and timbre through autographic mediums of paint, tinting of film strips, animated cut-outs, charcoal drawings, and drawn lines.
Particular early musical directors, such as Max Steiner, were criticized for vulgarizing the scene with precise synchronisation, by using music illustratively to respond to the action and mood of as film and by doing so translating musicality too directly the motion on screen (Birtwistle, 2010, p. 189). Bazelon characterises this compositional practice as redundant, as the viewer can already see the visual action, so the music’s amplification of the image becomes a distraction (Bazelon, 1975, p. 24). This is supported by the criticisms of two composers of the period Maurice Jaubert and Ingolf Dahl, who regarded mickey mousing as an inferior strategy, for its compositional replication of the path of the visual motion (Birtwistle, 2010, p. 190). For Jaubert, this criticism stems from mickey-mousing being external to a musical logic and responding to determinants in the scene, rather than to the structures and “established codes” of music itself (Birtwistle, 2010, p. 189).
Motion can be matched to sound via the movement of a body, moving objects, pulsating light, camera motion, and abstract changes that are related to the narrative (Branigan, 1989, p. 313). This is demonstrated in the first few years of sound-cartoons produced at Disney, particularly in the Silly Symphonies and early Mickey cartoons, which foreground musical performances as a key aspect of timing action and pacing through matching motion in cycles of repeated animation to the beat, bar and fixed number of frames (Jacobs, 2014, p. 60). Brophy suggests the term “fusion”, as an exacting descriptor of the sound and image track both having been “worked upon so as to distil each other, to effect a symbiotic relationship” (Brophy, 1994 Para 2). This symbiotic description of the effectiveness of synchronization is reflected by Jacobs, who encourages careful analysis of the complex rhythmic organization in early sound cartoons, particularly as they permit an appreciation of the opportunities which recorded music affords (Jacobs, 2014, p. 66). Jacobs suggests the inventiveness with regard to mickey mousing is due to the “clever synthesis of music and movement” rather than the “musical mimesis” of the on-screen action (Jacobs, 2014, p. 66).
In questioning what specifically determines the adhesion of sound to image, Birtwistle asks “what are the audio-visual parameters of mickey-mousing?” (Birtwistle, 2010, p. 190). He describes Mickey Mousing in a discussion of what determines the specific adhesion of sound and image, as being tied to the notion of morphology. This is described as “the sense of ‘shape’ as development over time”, or in focusing on the “morphological profiles of both sound and image” and their relation (Birtwistle, 2010, p. 190). The importance of observing the morphology of sound and image for Birtwistle, is for the possibility of conceptualizing “the material dimensions” of an audiovisual text which would not be possible using “significative modes of analysis” (Birtwistle, 2010, p. 190). The importance of this is in
27
the notion of morphology, which to a study of sound-image synchronisation brings a “recognition of the temporal nature of the cinematic text” and its “the parallel development of sound and image within a temporal frame, rather than any” (Birtwistle, 2010, p. 190).
28
The Classifications of the Taxonomy in the Project
In Chion’s discussion (1994), the function of sound is to add value to the moving image, or that the combination of audio and video creates a “total effect” that is greater than the separate modes (Wolfson, 2003). Chion discusses the term added value, that is present in “sound/image synchronism, via the principle of synchresis… the forging of an immediate and necessary relationship between something one sees and something one hears” (Chion, 1994, p. 5). Added value occurs when a distinct audio and video recording align to forge synchronization, whether synchresis in production has been fused accidentally or intentionally. Value has been added when the viewer has an impression in the “immediate or remembered experience” of the sound having emerged from what is seen is “already contained” within the image (Chion, 1994, p. 5).
The pairing of separate audio-visual recordings seemingly emanate as a contained unit and cloak the transduction between modes, as Hagood’s (2014) suggests, a “naturalised” moment of “synchresis obscures its own synthetic nature” (p. 110). The synchronisation of sound and image to create a synch-point reveals innumerable possibilities, as to accompany the moving images of a single body or face, due to synchresis “there are dozens of allowable voices”, some which add value more naturally than others (Chion, 1994, p. 63). Chion suggests even audio-visual combinations with seemingly no relationship may arise form “inevitable and irresistible agglomerations in our perception”, which he proposes by using forced marriage to discover synch-points (Chion, 1994, p. 63).
Chion discusses forced marriage as an exercise to discover synchronisation from chance alignments: by playing back unrelated films and soundtracks, or by removing the original accompanying soundtrack to discover unlikely synch-points between disparate video and audio content. Added-value is present in synch-points which align accidentally using the method of forced marriage or intentionally, though both entail a translation, of the formal components from one modality, that are reproduced or imitated in another at the same moment in the video sequence:
the expressive and informative value with which a sound enriches a given image so as to create the definite impression, in the immediate or remembered experience one has of it, that this information or expression "naturally" comes from what is seen, and is already contained in the image itself. Added value is what gives the (eminently incorrect) impression that sound is unnecessary, that sound merely duplicates a meaning which in reality it brings about, either all on its own or by discrepancies between it and the image (Chion, 1994, p. 5).
29
Kathrin Fahlenbrach proposes Chion’s Audio-Vision is a semantics of film, which unveils how meaning is divulged audio-visually through “autonomous semantic structures” (Fahlenbrach, 2005, p. 5). The acoustic and visual domains influence the other in moments, or “gestalts” of audio-visual synchronisation that are sensory and pre-symbolic. Fahlenbrach posits this as the central condition for synchresis: for them to be “experienced and interpreted as a necessary unit” is for the sound or the “acoustic elements” to be “continuous in filmed time” (Fahlenbrach, 2005, p. 74). The elements which comprise the visual track are perceived spatially, due to their being recorded discontinuously recorded and re-assembled as a montage. Fahlenbrach’s description of synchronous can be summarized as being inclusive of the relationship between the spatial and acoustic planes: of spatial information that occurs synchronously to “time-related information” which “are linked to a necessary semantic unit by the observer in an immediate fashion” (Fahlenbrach, 2005, p. 5).
The characterisation of synchresis as pre-symbolic is also observed by Betancourt, who elucidates his description of synchronisation: that as a starting point, the term synchronized for Betancourt is more than a “coincidence of simultaneous presentation”, or random audio-visual fusions which may emerge coincidentally from the exercise of forced marriage (Betancourt, 2017, Direct Synchronization Section, para. 1). In synchronization the audible realm becomes a “subordinate visuality”, by through demonstration rendering the “audible see-able”, rather than diminishing the “ordering role for sight” (Betancourt, 2017, The Role of Music and Theme Songs, para. 5).
Betancourt draws on Foucault in describing how synchronisation emerges via a pre-linguistic sensory encounter: stemming from the assortment that synchronisation occurs at a lower-level than the identification of letters and sounds, or those used for “the theorization and engagement with lexical forms” (Betancourt, 2017, The Synchronized Section, para. 9). Betancourt’s observation is a key point, that the innate immediate and inextricable nature of synchronisation is unlike written language, for having importance in “their initial, phenomenal engagements” (Betancourt, 2017, The Synchronized Section, para. 9).
Betancourt describes an instance of synchronisation as a “non-linguistic statement”, that derives from the audience’s experience to create meaning, as well as from the enunciation contained within it (Betancourt, 2017, The Synchronized Section, para. 10). For a statement to be formed, the image supplies context for reading meaning into a sound, or given “ordering schema, the identification of potential signifiers and their arrangement that is constitutive of the parameters that make interpretation possible”, as such, the meaning of the sound is always anchored by the image:
“the visual recognition and identification of letters, the distinction of which sounds are language and which aren’t—for sounds are independent of any emergent pattern (such as 30
“music”) that provide the central interpretation for the work. This experiential element’s primacy distinguishes these works from the linguistic; their analysis and interpretation can begin only through a process of identification and fragmentation that separates the continuities of experience for semiotic consideration” (Betancourt, 2017, The Synchronized Section, para. 9).
Synchronisation is a sensory encounter that is given meaning via a semiotic process, it is essentially “phenomenological” or distinguished from the linguistic, as Betancourt states the analysis “can begin only through a process of identification and fragmentation that separates the continuities of experience for semiotic consideration” (Betancourt, 2017, The Synchronized Section, para. 9). Betancourt importantly refers to the difficulty of discussing synchresis and reflects some of the shortcomings of this project for finding consistent definitions to categorize types of synchronisation.
Chion (1994) discusses the temporal and spatial affectivity of synch-points (Herzog, 2007, p. 30), which Fahlenbrach (2008) suggests are reflective of amodal qualities, such as “duration (e.g., long/short), intensity (strong/weak), position (above/below, central/ peripheral, close/distant)” (p. 96). Fahlenbrach refers to distinct modal aspects of the visual and audio as “image schemata as “force” or “balance” are projected on sound and pictures that create an audio-visual and emotional gestalt of the objects” (Fahlenbrach, 2008, p. 85). Simeon (as cited in Vernallis, 2004) suggests kinetic and syntagmatic relationships occur between audio and visual tracks occur at the level of the content itself. A Kinetic relationship refers to the changes between the tempo and the speed of the visuals, alternatively choreographing the visual may have a cinematographic basis, as the shot-type and movement of the camera reflect the rhythm of the music (Keating, 2013, p. 336; Vernallis, 2004, p. 181).
In narrative based screen-media, audiences rely on synch-points to support the story and mimic its “literal and emotional verisimilitude” (Deutsch, 2008, p. 96). Certain examples of synchresis display extreme congruence, as footage of sound being produced are used simultaneously in a musical structure. Warner (2017) suggests this directly illustrates the music as it necessitates the playback of a sound accompanied by the visual source at the moment of recording. This occurs in the music video for Coldcut’s Timber (IllPropaganda, 2012, Mar 8) as the sound of a buzzsaw repeats in the song to a video loop of a saw cutting through wood, depicting the visual source of the sound each time it occurs in the music. In the example, the audio and video reference the same moment of recording, as musical composition does not take for its basis the visual mode, extreme congruence in this musical capacity is rarely glimpsed in music videos, by rhythmically structuring the images and the synchronous-sound which had been recorded simultaneously (Kroon, 2014).
31
With specific production-roles dedicated to identifying the placement of a synch-point, as Kerner (1989) suggests, the standard rule of production is “if you see it, you hear it” (Kerner, 1989, p. 67). This approach to the image is called the ventriloquism effect, a term coined by Rick Altman which refers to a sound intensified the sense of reality produced from the images (Altman, 1980, p. 67). Altman simplifies this to a statement: “we see a door slam, we hear a door slam”, in which sound is used to anchor a “visual reference” of the door slamming “in our auditory sense system”, while he asserts nothing new is contributed by mimetic sound it “precludes any distantiation which might possibly be produced by the sight of a slamming door unaccompanied by its familiar sound” (Altman, 1980, p. 67). Altman provides a useful means of understanding how synchronisation occurs from a prescient mode following the trace of the other, as he characterises the soundtrack as a ventriloquist who creates the illusion of a dummy speaking by moving it to the words he speaks “whereas in fact the dummy/image is actually created in order to disguise the source of the sound” (Altman, 1980, p. 67).
In a moment of synchronisation either the visual mode or audible mode must be “prescient, foreshadowing some transformation that will soon occur in the other” (Vernallis, 2004, p. 190). As either the production of the video must anticipate the creation of audio or vice versa: the prescient track is the guide, foreshadowing the gestalt in the responding mode. Whichever video or audio exists first guides the imitation of gestures in the ulterior mode: either motion in the video track must either inspire the propulsion of the soundtrack, or the soundtrack alternatively catalyse the visual motion.
Helmreich (2015) uses the term transduction to describe the translation of one modality “across or between media”, noting that sound is transduced as it “undergoes transformations in its energetic substrate” (p. 222). The term transduction stems “from Latin transducere, “to lead across, transfer,” out of trans, “across, to or on the farther side of, beyond, over” + ducere, “to lead” (Helmreich, 2015, p. 222). In the process of traversing media, from antenna to receiver, from amplifier to ear, information undergoes “transubstantiations” that effects both its “matter and meaning”. Hagood (2014) adopts the term ‘transduction’ to mean “the process of turning a signal into "something else," as seen in the vibrational function of a phonograph's needle or the tympanic function of the eardrum” (Hagood, 2014, p. 100/101). Hagood (2014) uses this idea in his analysis of the synchretic combat Foley in Fight Club, elucidating on the “despatialised, detemporalised, abstracted violence” and its “visceral, haptic- auditory intervention into the alleged detachment of fleeting visual signifiers” (p. 110).
32
Points of Audio-Visual Synchronization
The relation between the audio-visual often occurs at points of synchronisation, or intermittent fusions of audio-visual parameters that for the remainder of the sequence are asynchronous (Chion, 1994, p. 167). Synchresis occurs during a synch-point, which Chion defines as moments in a timeline during which a “sound event” and “visual event” align, or “meet in synchrony”, similarly to a musical accent (Chion, 1994, p. 58). The key points of synchronization are crucial for conveying meaning, the placement of which define the phrasing of the sequence (Chion, 1994, p. 190). Synch-points allow for “dialogue underscoring” and the activation of “narrative functions” for musical ideas, that has been described in the previous section from theatrical tradition and the burgeoning stages of the silent cinema (Neumeyer, 2014, p. 17).
The task of underscoring the image with sound is assigned to the Foley artist and music editor, to bolster the elements that should be sonorous in the scene using the synchronization of sound-effects and music. Synchronization satisfies the utilitarian accompaniment of figure, ground and field sounds in the construction of a realistic environment. As such the role of the Foley artist is to follow the sync while reinforcing or replacing sounds, while also being to “sweeten” or enhance sounds that are outside “the jurisdiction of sound effects editorial” (Wright, 2014, p. 205). In accompaniment, the music editor’s role is to specify where the “music segments” will be synchronised with the images by taking spotting notes (Schifrin & Feist, 2011, p. 17). This is undertaken in a spotting session, or a preview of the film typically attended by the “the director, composer, film editor, music editor, the purpose of which is to choose places at which synchronization will occur, as in accordance with the director’s instructions (Schifrin & Feist, 2011, p. 16).
As synch-points are spotted throughout an audio-visual sequence in fleeting bouts of fusion, the viewer is “subjected” to synchresis and may find “it difficult to remain cool and detached” (Beller, 2011, p. 110). Synch-points may pass unexamined due to the intervention of added value, as is typical of fight sequences in martial art films which spot a sequence with “whistles, shouts, bangs, and tinkling” (Chion, 1994, p. 11). Jacobs (2014) argues that “the crashes, bangs, and toots typical of the cartoon effects track, as well as the abrupt rhythmic shifts characteristic of cartoon scoring, help to delineate audio-kinetic filmic units that are akin to the rhythmic delineation of the phrase in music” (Jacobs, 2014, p. 83). The spotting of synch-points throughout a video timeline are used to enact various functions, which Chion (1994) enumerates:
• As an unexpected double break in the audio-visual flow, a synchronous cut in both sound and image track.
33
• As a form of punctuation at the end of a sequence whose tracks seemed separate until they end up together (synch point of convergence).
• Purely by its physical character, for example when the synch point falls on a close-up that creates an effect of visual fortissimo, or when the sound itself is louder than the rest of the soundtrack.
• Affective or semantic character: a word of dialogue that conveys a strong meaning and is spoken in a certain way can be the locus of an important point of synchronization with the image (Chion, 1994, p. 59).
Sonnenschein suggests that anticipation to synchronisation can be anticipated while not being fulfilled, which he terms a false synch-point (Sonnenschein, 2001, p. 167). For instance, as a gun is pointed towards someone and the gunshot is heard over the image following a cut, in such instances Sonnenchein suggests we fulfil ourselves as viewers as the synchronism doesn’t exist in the construction of the sequence. The engagement with the content in this way suggests more “intimate” engagement with the action in which the viewer is “participating internally” (Sonnenschein, 2001, p. 167). To further burrow into the taxonomy, the next level of discussion the synchretic envelope, to suggest the differences between the durations of synch-points.
The Synchretic Envelope of a Synch-Point
From sound theory I have borrowed the term ADSR envelope, to illustrate there are short and long instances of audio-visual synchronisation: those which are short, accented fusions of synchronization in a timeline and others in which the synchronism is sustained (Breslin, 2011, p. 140). In the early 1960s, the attack, decay, sustain and release (ADSR) module was proposed to Bob Moog by the experimental composer Vlademir Ussachevsky, as a suggestion for a module which would provide a variety of uses for musical composition (Aikin, 2007, p. 50). The term envelope refers to a combination of level and time settings: while the attack, decay and release parameters determine time, the sustain parameter determines the level. Aikin expands on this description in regards to the creation of sounds using envelopes on a keyboard or synthesizer:
“The rate at which it rises is controlled by the attack time parameter. At the end of the attack, the envelope falls back to the sustain level. The rate at which it falls is controlled by the decay time parameter. It then stays at the sustain level for as long as the key is held down.
34
When the key is released, the envelope falls back to zero at a rate determined by the release time parameter” (Aikin, 2007, p. 50).
In implementing the idea of the ADSR module with relation to synchronisation, for each instance of a synch-point, the envelope is a measure of the level and duration of the audio-visual synchronism. Despite which values of the visual and audible are fused, the reason for adapting the term envelope to a taxonomy of synchresis is to refer to the synchronised motion of separate formal elements of the audio-visual, which are boosted or attenuated (rise or fall) in synchronisation.
Vernallis maintains that for Chion “all sounds consist of an attack and a slight fading resonance” which are “oriented in time in a precise and irreversible manner” (Vernallis, 2004, p. 177 and 178). The term sound envelope is defined as “the shape of the sound wave”, of which there are four elements: “how quickly does a sound wave peak at its maximum amplitude (attack)? How quickly does it back off from that peak (initial decay)? How long does the sound last (sustain) before it drops out of hearing (decay)?” (Breslin, 2011, p. 140). In keeping with this description of a sound’s envelope, the envelope of a synch-point refers to the distinct elements of the audio-visual in synchronisation: maximum intensity (attack transient), a period of time in which the intensity is maintained (sustain), and the period for it to recede to imperceptibility (decay).
As each synch-point is comprised of rising and falling values, the shape of the envelope is a visual representation of the duration and intensity of audio-visual elements in synchronism: of the sequence’s lessening (attenuating) and increasing (boosting) of audio-visual elements across a video timeline. Unless sync-sound was recorded, or if both the images and sound were recorded simultaneously, the production of synchronization in post-production demands either the audio or video is prescient, that is a leading and an imitating mode existing in advance of the other.
Synchretic Parameters and Operations
Despite the impression of a unified and inextricable audio-visual gesture, synch-points are formed via separate audio and visual parameters aligning at the same moment in a timeline, or of synchretic parameters that have changing intensities over time, which can be understood as being contained in a synchretic envelope. As the audio and visual synchronise, a transduction occurs from the envelope of the presciently created mode, which catalyses the imitating mode: either the videotrack is created first and the audiotrack follows, or the audiotrack is created first and the videotrack follows. To define my use of the term, I suggest a visual parameter is any element of the moving image that may be altered
35
in isolation: stemming from the camera movement, the mise-en-scene, whether animated or live- action, cinematographic or software generated.
Concurrently, an audio parameter is any element a sound which can be varied in isolation, whether a recorded sound, a noise produced by an instrument, a sound effect or from the addition of audio effects to augment a sound. Synchronization at the level of parameters illustrates the innumerable ways audio-visual fusion can occur, as for a synch-point to form at least one audio parameter and visual parameter must simultaneously correspond in the timeline within a synchretic envelope. Identifying and discussing which parameters comprise the envelope is problematic, due to the inextricability of their occurrence at a detailed level of synchronization, which seldom occurs between a single audio and visual modal parameter, but between multiple audio-visual modal parameters. As a synchretic envelope emerges via the changes of distinct audio parameters (such as pitch, timbre, loudness, distortion, delay, reverb, tremolo,) and visual parameters (such as brightness, saturation, height or depth, distortion, zoom, blur), the parameters of the audio or visual composition are separate, though give the impression of being unified from their raising or lowering of their prominence at the same point in time.
The aim of producing a synch-point might begin with a recorded sound as the prescient gesture, to be translated or transduced into a visual mode. In synchronising the parameters of brightness and pitch: as the sound ramps from a deep warble to a high pitch squeal, the image might begin underexposed and gradually become overexposed. As more than a pair of parameters are synchronised, the inextricability of the elements in synchronization complexifies. In addition to a pitch rising, for instance, a sound could gradually distort, which might be translated with the application of a kind of visual distortion. While the upward trajectory of the pitch may be synchronised with the object moving laterally along the Y-axis of the screen or moving upwards in scale, the gradual distortion may also be translated with a visual parameter, such as the addition of grain or camera shake to match the trajectory of the distortion’s application. The use of DAW (digital audio workstation) such as Ableton Live, and the non-linear editing software used for the captured audio and video in the artefact have allowed for the animation of audio and visual parameters via keyframes.
36
Figure 1: The automation of a plugin in Ableton Live.
In this project, I have explored four different synchretic operations: collision, cutting, dynamism and transmutation, which each classify a different kind of audio-visual synchronization, or of differences in how the duration of the intensity of the synchretic parameters. Instances of collision and cutting are produced via a short-envelope, which as short bursts of synchronization in their fleeting occurrence and accented form are the opposite of the long-envelope operations of dynamism and transmutation. Instances of collision denotes a point of synchronization via the surfaces of two objects colliding to produce a sound in synchronization, while cutting a sound is produced in synch at a cut in a sequence. This is the opposite of the long-envelope operations, which in instances of dynamism follow the trajectory of movement in synchronisation, while in transmutation a shift occurs from within an object that in synchronisation with a musical cue or sound effect produces a synch-point.
Music Video as a Site of Audio-Visual Synchronisation
For theorists and critics alike, the music video flaunts the line between art and commerce, to which there is no comparable schizophrenic art form (Feineman & Reiss, 2000, p. 10). Music videos are a media hybrid, a blending of artforms which tread an intermedial line between programming and the commercial (Aufderheide, 1986, p. 57; Dickinson, 2007, p. 13; Feineman & Reiss, 2000, p. 10). Research illustrates music video’s ancestry in modernist and classical art tendencies, that are considered popular culture (Fiske, 1987; Kaplan, 1987; Taylor, 2007), as well as a tool of advertising and communication (Allan, 2015; Tetzlaff, 1986). As a post-modern form characterized by the diminishment of narrative and character development to favour fragmentation and emphasize emotion, music videos create worlds based on simulacra and metatextuality, that interweave their own content with other cultural texts (Feineman & Reiss, 2000, p. 13; Peverini, 2010, p. 146; Sánchez-Olmos & Viñuela, 2017, p. 3635).
The fragmentary nature of music video has been assigned to the shifting and kaleidoscopic relationship between sounds and moving images, which present an inconsistent hierarchy between 37
sounds and images (Lapedis, 1999, p. 367). Commentators have suggested the editing contrasts the style of linear narrative, or of shots organized into sequence with clear spatial, temporal and causal relationships, that are instead attached to the musical structure for an audience member to quickly engage with (Björnberg, 1994, p. 51 & 52; Frierson, 2017, p. 335). Jones proposes this narrative fragmentation from traditional representations of time and space is cohesive though not coherent, for which he assigns the term digital narrative to refer to information being presented in discrete steps (S. Jones, 1988, p. 17).
The primacy of invoking emotion rather than story has been proposed as a reason the fragmentation of music video narrative, which vie for constructing a feeling state which appeals to viewer in clusters of dreamlike imagery (Dancyger, 2007, p. 166). These combinations are narratively “developmental” and “disjunctive”, which it has been suggested is due to the emotion being so closely related with a particular piece of music (Dancyger, 2007, p. 168). Ayeroff (2000, p. 7) argues that throughout the short history of the music video, selling has become increasingly competitive, sophisticated and interesting, as a music video appeals to the viewer with the fantasy of a shared experience of “a wondrous leisure world” (Aufderheide, 1986, p. 62). The effectiveness of the advertising is in part due to the focus on creating an audio-visual fantasy, which increases the effectiveness of the promotional form by not being acknowledged as such (Englis, 1991; Korten, 2015, p. 155).
In their constant reinvention and varied permutations, the raw materials of a music video are comprised of the intermedial and voluble well of commercial popular culture, that abolish the boundaries between fantasy and reality (Aufderheide, 1986, p. 77). These forays “into the arbitrary” operate by triggering “mood states”, as the elements of the moving image endeavour to explore their relation to music (Bonde Korsgaard, 2012, p. 2). Kinder notes that most rock videos resemble a disparate chain of images, that resemble dreams through their discontinuities in space and time structures (Kinder, 1984, p. 3). The beat of the music is accentuated via the “pulsating kinetic rhythms of the visual montage”, which from being so closely linked to the music and lyrics provides a “unifying identity” to the discontinuous videotrack (Kinder, 1984, p. 3).
The strategic practice on which a music video relies is comprised of the editing process and the mise- en-scene, both of which filmically enunciate the soundtrack (Peverini, 2010, p. 137). The form disseminates music and circulates the performer, which often present the song as emanating diagetically as the song is performed, and appear to emanate from within the world of the video (B. Allan, 1990, p. 4). Vernallis (2004) suggests the image showcases the song while being freed from its linearity, as the image connotes visually, fulfilling what the other cannot through “the locked gate of semiotic difference” (p. 194/195). With the aim of interpreting musical cues visually, the image relies
38
on the compositional elements of the song’s “shape, texture, and flow” (Vernallis, 2004, p. 175), which through synchronization can elicit a stylistic spontaneity that is constructed to reflect the characteristic of the liveliness of a musical genre (Bolter & Grusin, 1999).
The style of editing of a music video emphasizes fragmentation, which favours a non-linear approach to narrative via the visual expression of musical cues. Rather than a focus on story events and characterisation, the traditional narrative editing is replaced with a “multi-layered approach”, the focus is a “place, feeling, or mood” (Dancyger, 2011, p. 165; Frierson, 2017, p. 335). As the dramatic flow is abstracted and musical signals are translated to moving images, the imagetrack needs little justification than the connection produced between the formal properties of the instrumentation, between their properties “at the level of rhythm, pitch, duration, and tone colour” with formal visual properties of the moving image (Bonde Korsgaard, 2012, p. 3; Diederichsen, 2011, p. 121; Keazor & Wubbena, 2010, p. 23).
Chion (1994) suggests collision editing is particular to music video, which enhances the linear and rhythmic dimensions of the image while leaving the spatial dimension impoverished (Chion, 1994, p. 150). This kind of editing is not constructed to serve the purposes narrative, as Caston argues that by responding to musical features, such as rhythm, timbre, melody and form the editing of music video exceeds that of narrative film (Caston, 2017, p. 3). This is not necessarily a view widely held by the majority of editors, though it is often a key aspect to the form of media production and the task of drawing out musicality. As Barbara White suggests recounting collaborating as an editor working with composers, who would hesitate to coordinate their music to the film (White, 2006, p. 65). Certain sounds may be drawn into prominence which direct the viewer to listening as the visual mode reflects the scalar or pitch variations in melodies, or in synchronization with the rhythm or meter of the song (Warner, 2017, p. 168). In discussion of the depiction of musical qualities, Kaplan (1987) likens the direction of the of the MTV viewer’s attention to a baby vis-à-vis the mother’s face, in which the spectator is locked into a one-way process with the performer and is rarely able to process and decode information due to being taken up by the fleeting nature of visuals and flow of the music (p. 91).
39
Commercial Advertising as a Site of Audio-Visual Synchronisation
Pollay states the aim of advertising is to preoccupy individuals with the idea that the solution to problems and happiness is attained via commercial goods or services (Pollay, 1986, p. 21). As a contributing aspect to the service of a message, the synchronisation of moving images with verbal and musical cues are key to the signification inherent to commercial advertisements (Scott, 1990). With a quarter of broadcast time dedicated to advertising, through accretion and reinforcement, values are presented as commonplace, executed by professionals and consumed by an audience that has become detached from traditional cultural influences (Bergman, 2018, p. 161; Pollay, 1986, p. 21) . Josephson describes the process of advertising as creating desire for a product, that has an aim which is to “cause us to act”, particularly towards being involve in the “economic life” of a nation by purchasing productions and continuing “the momentum of industrial production” (Josephson, 2015, p. 152). With the aim of influencing the purchases of an individual, advertising has become increasingly concerned with the persuasion of personal values, by attaching a social meaning to material goods and crossing the two needs so that neither is fulfilled (Williamson, 1978, p. 23).
It has been suggested advertising is the translation of a brand “expressed as a benefit” or a “value proposition”, in the form of a message delivered via media to the consumer (Calder, 2008, p. 1). The connection to other domains of culture endows advertisers to suggest covertly, how through consumption, one is able to satisfy their desires and aspirations (Beasley & Danesi, 2002, p. v; Jhally, 2002, p. 328). The desire to attain happiness is met with market-oriented solutions via the purchase of goods, which conflate social domains with material domains. As such, the promise of happiness is offered via the suggestion of balanced social relationships and leisure-time which are not fundamental to the goods themselves (Marchand, 1985, p. 348). In doing so, ideals and values are subordinated to the attainment of material satisfaction through the purchase of goods (Schudson, 1993, p. 220).
Scott suggests audience responses are a learned and “socially situated, communicative experience” that are grounded in cultural convention (Scott, 1990, p. 227). Like art, advertising is more commercially effective by drawing on specific elements of the world with which the target audience is already familiar in the production of new meanings (van Niekerk & Conradie, 2016, p. 236). In drawing upon and rechannelling prevailing social and cultural values to depict the attainment of material satisfaction (Schudson, 1993, p. 221). Each advertisement consumed is thus framed by those preceding it, as interpretations and communicated meanings are a culmination of the listener’s media experience, both in relation to “the immediate temporal experience of the particular stimulus” or the
40
message syntax, and in relation to a listener’s matrices of associations with similar musical stimuli from previous advertising (Scott, 1990, p. 227).
Synchronising moving images and popular music can be an expensive process which requires a synchronization license, in which a fee is required to pair a set of images with a certain song. Large fees are paid to associate a brand with a piece of music, to enhance attitudes towards brands and of sales (Craton & Lantos, 2011, p. 396). Depending on the prominence of the song in the material, on the intended site of broadcast and the length of time for which an advertisement may be broadcast from $50,000 to $200,000 for annual usage (Allan, 2015, p. xiv). Research has focused on the ability of music to enhance an advertisement’s effectiveness by attracting disengaged listeners (Craton & Lantos, 2011, p. 396; Olsen, 1994; Stout & Leckenby, 1986), and of the effectiveness of music’s influence on mood states of a consumer (Alpert & Alpert, 1990; Gardner, 1985). Further research suggests that music influences behaviour, as in the work of Bruner, particular genres and styles have been associated with consumer choices (Areni, 2003, p. 162; Bruner, 1990, p. 94).
With the aim of intimately appealing to a viewer’s identity, communication in advertising is a one-sided exchange, reflected in Barthes’ characterisation that “all advertising is a message” (Barthes, 1988, p. 173). The commercial format assumes one-sided techniques of communication, in which responsibility can be waived, as Baudrillard suggests, what characterises the mass media is the fabrication of non- communication, that forbids response and exchange: “if one accepts the definition of communication as an exchange, as the reciprocal space of speech and response, and thus of responsibility” (Baudrillard, 2010, p. 52). As synch-points are placed, or “spotted” through key moments in a commercial advertisement, their fleeting occurrence is intended to instil succinctly and communicate meaning with brevity, the emotional power of which is strengthened by an audience’s members rarely committing to processing deeply each audio-visual information (Feng & Morris, 2016, p. 193).
Michael Schudson suggests advertising is an interruption, that is accepted and watched inattentively for the simple reason that “advertising is propaganda and everyone knows it” (Schudson, 1993, p. 4). While not all synch-points spotted in a commercial advertisement are deciphered by a viewer, the fleeting convergence of “musical, visual, and verbal” cues is purposive, and therefore should be read as an indication of the communicator’s “character or intent” (Scott, 1990, p. 227). Jhally deems this mode of communication the commodity information-system, which relies on moving images in addition to the increased rapidity of non-static images that constitute and dominate public space, as the speed of a sequence disallows the viewer to linger on the flow of images and sounds, the rapid playback of images has the consequence of immersing the viewer in the message (Jhally, 2002, p. 333). The
41
synchronization of musical, verbal and visual cues are investigated in the It’s The Delmer Brothers moniker, in which I have created two music videos in the style of vintage commercial advertisements.
A study by Krishna reports how advertisers can engage and appeal to consumers with sensory triggers, the advantage of which is to “self-generate” a favourable view of brand attributes, rather than have them be provided (Krishna, 2012, p. 341). Moving images in an advertisement must also be carefully paired, as the “sound emitted by products” is significant for impacting brand perception, as for instance in a spokesperson for a brand (Krishna, 2012, p. 341). A spokesperson’s voice may “lead to inferences” about a product’s features, despite how illogical drawing the inference may seem (Lowe & Haws, 2017, p. 333). Similarly, Preston observes that linguistic associations used in depiction of a product which are nondeceptive legally, such as associating a product or service “with sociological or psychological values that it does not provide literally, objectively” (Preston, 1996, p. 151).
Of its nature, advertising collapses the boundaries between the product and everyday lives (MacRury, 1998, p. 28). Researchers have suggested how meaning can be transferred to a product, relating to consumer goods carrying with them cultural meaning and identity (McCracken, 1986; Sahlins, 1972; Schwartz, 1967). A body of enquiries have investigated whether a product embedded in a scene is an effective auditory and visual stimulus to affect brand recall (Gupta & Lord, 1998; Russell, 2002). This exchange is where meaning is created, as material objects that can be purchased are conflated with the representation of non-material needs (Williamson, 1978, p. 23). As a site for exploring cross- modality, marketing provides a unique site of enquiry for exploring the transfer of meaning from the audio-visual representation to the product itself: “from one stimulus (advertisement sound) to another (physical product) by symbolic association” (Lowe & Haws, 2017, p. 333).
Work in the area of “sensory marketing” focuses on cross-modal correspondence, or the perception of a stimulus received in one sense modality and experienced in another sense. Researchers have interrogated the relationship between the size of an object (small or large) with the pitch (low or high) of an acoustic stimuli (Lowe & Haws, 2017; Spence, 2012). Lowe and Haws observe how by virtue of association the dimension of a sound’s pitch can effect consumer beliefs regarding a products size, in the investigation of a sound being “intrinsically associated with other sensory characteristics” that are not sounds (Lowe & Haws, 2017, p. 331). As marketing is a unique context for the exploration of cross-modal or multi-sensory integration via the sound to the physical product via association, the aim of its use for marketers is implant subconscious triggers which “characterize consumer perceptions of abstract notions of the product”, in order for the consumer to self-generate a favourable attitude towards a brand (Krishna, 2012, p. 332).
42
Research by Evans and Treisman, similarly investigated the relationship found correspondence between pitch and the visual features of vertical location, size and frequency (Evans & Treisman, 2010). Research has been produced exploring the cross-modal associations between the dim/bright and dark/light colours to low/high pitched and soft/loud sounds (Marks, 1989; Marks, 1987). Related to synchresis are research projects concerned with the speeded-classification paradigm, which as Lowe and Haws suggest, observe the speed of a participant classifying a visual stimulus as large or small, while a low or high-pitched sound is played alongside it (Lowe & Haws, 2017, p. 332 and 333). The method of enquiry for the majority of these studies, is to expose a participant simultaneously with visual and acoustic stimuli, in which they judge the temporal order the stimuli (Bien, Ten Oever, Goebel, & Sack, 2012; Parise & Spence, 2008).
The use of synchresis in an advertising context is reminiscent of the advertising strategy called chunking, in which information is grouped as multiple aspects of the advertising message connote a similar message. Harris et al. suggest a chunking strategy using a sealant named Water Seal, that “once used a logo of a seal (animal) splashing in water in the middle of a seal (emblem)” (Harris, Sturm, Klassen, & Bechtold, 1986, p. 4). This creates a “unitary memory representation”, via the combination of the animal, the function of sealing, the representation of an emblem and the sound (/sil/), which via the “highly imageable homophone (the animal seal)” allows for multiple “avenues of retrieval” (Harris et al., 1986, p. 4).
43
Medium-Specific Noise in the Production of the Artefact Submission
During the production of the artefact submission, various audio and video recording media were used for the “distinct sensory qualities” that are produced directly via their “materiality and technology” (Newman, 2014, p. 32). The distinct sensory qualities that emerge by using a particular recording medium or combination of media are referred to herein as medium-specific noise, which are discussed to address the use of noise as an aspect of audio and video recording (Fetveit, 2013, p. 189). Rather than being eradicated to preserve unobscured recordings, noise and distortion provide the basis for particular synchretic experiments in the artefact. This section addresses the semi-professional and amateur equipment used in their production, that from the incorporation of particular recording media has resulted in the presence of various kinds of noise and distortion that is observable in the works.
As the recorded sound and video in each production each have varying kinds of noise and distortion, this section introduces the analogue and digital recording technology used, to address the process of recording and shaping noise and distortion to produce synch-points. Gordon (2010) expands on McLuhan’s (1964) definition of a medium as any extension of a human being, so that as clothing extends skin so too the computer extends a human being’s central nervous system, as does housing extend the human body’s temperature mechanism and the bicycle and stirrup extends the foot (Gordon, 2010, p. 107). Subsequent media innovations “repackage” their forerunners, as McLuhan contends writing contains speech, the written word contains print, or that of print “the content of the telegraph” (McLuhan, 1964, p. 8).
Rather than one medium being in in competition with another, they are relationally situated to other media within a constellation, or “a wider ecology of technologies, representations, and meanings” (Newman, 2014, p. 3). By inheriting and improving upon their ease-of-use, the functions of older media are sutured into newer media and as such “content” of a medium is comprises subsequent innovation (Kress & Van Leeuwen, 2001, p. 90). Given this, it is not only the noise that a particular medium is used for but also its “cultural status” in a specific historical context (Newman, 2014, p. 3).
The audio and video recordings which comprise the artefact submission both incorporate analogue and digital media, the significance of which pertains to the way recording and encoding is performed (Frigerio, Giordani, & Mari, 2013, p. 455). The term digital regarding the musical recordings refers to the recording of an audio signal using a computer software via a USB audio-interface, or a digital capture card when in reference to the transcription of analogue video into a digital form. The digital media refers to video recordings produced using a Canon 5D DSLR camera, Black Magic Camera, 44
Digital HDV handicam, Mini-DV, and an I-phone 4S camera. The analogue photo and video recordings were produced using a VHS videotape, a VHS-C videotape, a 35 mm SLR and Super 8 film camera.
The images in the artefact were captured using polaroid photographs, magnetic videotape, digital video (DV), digital SLR cameras, high-definition digital video (HDV), 35mm film and computer- generated images (CGI). Used in conjunction with the visual recording media, the sounds in the project were produced using analogue and software instrumentation, comprising synthesisers, microphones, bass guitars, electric guitars, drum kits, compressors, equalisers, analogue and digital samplers, virtual studio technology instruments (VSTi), magnetic cassette-tape recorders and a quarter-inch reel-to-reel tape recorder. These various recordings are captured are arranged in a DAW (digital audio workstation) in Ableton live and sequenced to images in a non-linear editing software using Adobe Premiere Pro.
The means by which information is transcribed and the varied formats with their differences in quality and process bear an effect on the capturing images and sounds. Analogue recording technology is used to invoke the “warmth, authenticity and wholeness” that is characteristic of previous decades of musical and video production (Hausken, 2013b, p. 196). This is evidenced in the separation of artistic monikers by the recording formats and genres which imply cultural production of a particular era: as quarter-inch tape is used to be indicative of music recorded in the 1970s, videotape and the sound of cassettes are used to convey the 1980s cultural production, while the rigid and digital quantised rhythms of a digital recording technology is indicative of the 1990s and the 2000s. In the aim of establishing a palpable sense of “the real, presence and location”, the media that is sonorously characteristic of a particular musical idiom establishes a “nostalgic relation” to the past through the quality of the recording itself (Hausken, 2013b, p. 196).
Noise is used in the submission of music and music video, as well as in the re-photography methods of producing the album art, to imply the authenticity of the artefact submission as an archive of reissued music from previous decades. It is intended that noise and distortion inherent in the recordings by the use of particular medium set up a relation to the past, that as Fetveit observes: the nostalgia that is conveyed by noise do not return us to a past, but set up a relation to it that is premised on the evocation of an older medium by way of its medium-specific noise (Fetveit, 2013, p. 208). As Encarnacao observes, it is this aspect of the production of the recordings that frames and carries the “meaning itself”, as I have hoped to achieved by the media and their evocation of a previous era of cultural production (Encarnacao, 2013, p. 7).
The obscuring of the clarity of a sound or image is typically characterised as an undesirable aspect of the work, although it may be used expressively as an aesthetic device for “attaching a kind of tactility” 45
to a recording (Hausken, 2013b, p. 191). The analogue cassette and quarter-inch tape recording methods in the production of artefacts are “subject to entropy” (Rodowick, 2007, p. 113), as are the videotape recordings susceptible to the addition of interference and noise (Breslin, 2011, p. 135). As analogue information is duplicated via another analogue copy, the process of reproducing the information “preserves the isomorphism of its source” with the caveat of degrading the quality as each copy, or a generation is produced (Bond, 2012, p. 20). In addition to a precise editing workspace, digital recording allows for a dramatic increase in sound quality that is free of hiss, in which an unlimited number of tracks may be layered, that as Partyka notes is a significant difference from the “hiss-prone” four track (and eight track) multitrack recorder as have been used in the production of the artefact component (Partyka, 1999, p. 50).
The analogue process is transcribed or imprinted onto a “continuous physical object” such as celluloid, magnetic tape, canvas or clay (Guter, 2010, p. 7). In contrast, the digital process is characterised by a discrete form, whether it has originated as computer generated images or sounds, or if it is “converted from various forms of analogue media” (Guter, 2010, p. 7). The artefact submission demonstrates recordings using an analogue process which have been converted into a digital form, to retain their expressive noise (for instance, the sound information has when it is recorded on a cassette), while allowing for the duplication and rearrangement that is facilitated by a DAW. The magnetic tape recordings in the artefact are used for their expressive aesthetic effect, as the recordings are transferred from a cassette multitrack machine into the computer.
The process of “analogue transcription and digital conversion” is undertaken in the rearrangement and attenuation of analogue recordings using the advantages of a digital workstation, allowing for experimentation with the sequence of sounds, which could be redacted and returned to their original state (Rodowick, 2007, p. 122). Manovich describes the process of “new media objects” as being created and arranged using computers in numerical form, which takes part in the stage of “sampling and quantisation” (Manovich, 2001, p. 28). Data is first sampled at regular intervals and subsequently each sample is quantified as a numerical value (Guter, 2010, p. 7; Manovich, 2001, p. 28). The data’s status as a discrete form, as opposed to continuous, is randomly accessed rather than sequential, it is mutable in that each sample is replaceable “by any other sample”, that in their storage in a computer’s memory are retrievable and reproduceable without degrading the information (Guter, 2010, p. 7).
Given the status of the audio recordings as mutable computer data, the recordings are organised into songs by listening and arranging recordings in the composition, as Breinbjerk notes given the age of digital media and audio being subjected to the language of code, the approach to music and its compositional practices are less centred around musical literacy, but products which are wrought from
46
listening to sounds (Breinbjerg, 2007, p. 32). The case for the contemporary composer, which is acknowledged in Manovich’s explanation of media’s status as computer data, as the creator employs variations of similar techniques, to “copy, cut, paste, search, composite, transform, filter” the audio- visual material (Manovich, 2001, p. 118). What is reflected is this project is the effect of change technologically confuses the “usual distinction” between the roles of a sound engineer and musician (Frith, 1986, p. 265). As Simon Frith suggests, the unequal distribution of technological power has empowered musicians, that far from being oppressed by it has made musicians more inventive in their engagements with technology and attempts to control their own sounds” (Frith, 1986, p. 273).
As a general rule, in analogue recording, the less deterministic the recording is, the more likely noise will emerge from the complexity of the signal path (Hartmann, 2013, p. 179). Noise can be added from the use of magnetic tape from the “granular, or particulate” coating material and the “magnetic domains within it” (Hood, 1999, p. 27). Noise can also arise from issues with the recording of the sounds themselves, from the lesser problem of circuit noise, the choice of the tape and “the electronic amplification and signal handling circuitry associated with it” (Hood, 1999, p. 27). Hood observes the similarity between the problem of noise on magnetic audio tape and of graininess in a photographic process, that in both are “related to the distribution of the individual signal elements” and increase in severity “as the area sampled, per unit time, is decreased” (Hood, 1999, p. 28). In addition, from magnetic tape as a medium noise may arise due to the surface characteristics of the tape, known as modulation, bias and contact noise.
The prominence of semi-professional and amateur recording equipment in the artefact has resulted in a combination of noise and distortion in the recordings. The distortion and the noise in the artefact are used to produce experimental deviations which can be captured and used for the purposes of synchronisation. Treharne and Walker describe the process of distortion as a non-transubstantive form of the original, that through the process of being distorted becomes “warped, misshapen, skewed, shrunken, amplified or simulated” (Treharne & Walker, 2017, p. 1). As Kendall, Haworth, & Cádiz observe, it is emblematic of the “auditory distortion products” throughout music history to emerge as “happy accidents” and less as “directly controlled musical material” (Kendall, Haworth, & Cádiz, 2014, p. 8). Similarly to the visual distortion throughout the works in this project, the capture of errors and unintended gestures would often arise from experimentation with the technology (Kendall et al., 2014, p. 8).
47
The Working Processes Comprising the Artefact Component
The project as it is structured as the output of a record label allows for the exploration of a fictitious body of work produced by multiple authors, by varying the production methods and technologies to suggest music which is characteristic of an era of production (the 1960s, 1970s and 1980s). As each moniker varies the musical genre and combination of recording media, they also form the basis for differences in the method of producing synchresis. To reduce the submission of artefacts, several hours of recorded and sequenced music have been omitted, the music which has been included in the submission is comprised of four extended play (EP) albums and four long play (LP) albums, which are a combined 157 minutes in duration.
The distinctions between monikers in the artefact submission is to emphasise separation between the varied approaches to the synchretic operations of collision, cutting, dynamism, and transmutation. With the aim of questioning the idea of “authorial production” and varying the approach to synchresis at the level of production by creating multiple identities that would supply a varied groundwork for a visual counterpart, each moniker is representative of different recording methods, media and musical styles (Partington, 2008, p. 10). The design of the accompanying album covers are not directly concerned with synchronisation, though they reinforce the idea of multiple authors, as each image has the appearance of being distributed on a compact disc, vinyl record or cassette.
It’s the Delmer Brothers!—Commercials 1975 – 1981
The five songs and two video commercials in the It’sThe Delmer Brothers! submission explore synch- points that are characteristic of commercial advertising, in which musical cues and voiceover narration guide the spotting of dynamic, collision and cutting. There are five songs written and recorded, as commercial jingles for each product: Cindy Doll a doll, Rainbow Shoes a brand of sneakers, Seed Brew a brand of beer, Sam’s Little Patriots action figurines and L’Efforel. Two of the five It’s the Delmer Brothers! songs have adjoining music videos, whereas Cindy Doll is a commercial for a toy doll and the L’Efforel is a commercial for cologne.
The music videos and songs recorded submitted as part of the It’s the Delmer Brothers! moniker are made to be reminiscent of vintage commercial advertisements, which explore synchronization between the instrumentation and singing of a musical jingle and voiceover narration with text and graphic animation. The music videos are made in the style of vintage commercials, of which the audio tracks were recorded presciently on multitrack cassette with the intention of producing fixed locations at which voiceover and musical cues would occur. In imitation of the production of vintage commercial advertising, the two videos are experiments with presciently recorded songs which were fixed due to 48
being recorded on cassette, that supply the guiding lyrics, and voiceover in reflection of the functions of the advertised product.
The Fed-Up Kids—Campus Slump OST (1981)
The ten songs and two music videos submitted in the The Fed-Up Kids moniker focus on naïveté as a thematic groundwork for which to align dynamic, cutting and collision synch-points. The output of The Fed-Up Kids moniker is characterised by short songs produced in the style of a 1980s pop group, which are structured in rondo form (A,B,A,B,C), comprising a verse, chorus and outro. These are driven by thematic motifs of self-imposed loneliness and indulgence, isolation, schoolyard drama and rebellion. To convey a childlike and naïve theme in the songs, The Fed Up Kids moniker was initially envisioned as a fictional 1980s coming-of-age film, with fed-up being an allusion to grown-up, in referring to myself recording the children in the title as an adult. In the accompanying extended play (EP) collection of nine songs, each of the vocal tracks has been recorded using a pitch-shifted vocal style, to represent the group as being comprised of American teenagers.
The instrumentation used in The Fed-Up Kids songs are saw wave synthesisers, distorted guitars and drum machines, pitched-up vocals, keyboards and an electric guitar. The songs were recorded prior to the capture of video, with a focus on simple lead melodies reinforced by multiple instruments and vocal tracks simultaneously. As the musical recordings were created presciently to the visuals, the rapid and clear melodies supported by the mix of instrumentation in the two music videos The Heel and School are experiments in collision, cutting and directionality. The music video for The Heel explores Mickey Mousing between unbroken footage of fast-forwarding cars and pedestrians in synch in an in synch to a clearly reinforced melody. While in the music video for School synch-points align between handheld footage of the motion of vehicles, buildings and flowers to melodies that are simultaneously reinforced by the ensemble of recorded instruments.
Peter Bowman Jr.—City Nights (1984)
The twelve songs and two music videos submitted of the Peter Bowman Jr. moniker suggest the output of a 1980s male solo artist. The music of the submission is characterised by theatrical, synthesiser-driven pop songs with a deeply pitched-down vocal style, to be reflective of being written and recorded in the style of new wave pop music. The songs are thematically and stylistically illustrative of a televisual, mediated representation of a male identity, in which the aesthetic style and lyrical preoccupations of the songs are reflective of cinematic themes. The use of sound effects to reflect various cinematic tropes of film noir, of themes of romance, loneliness, crime and of the city at night.
49
There are two videos submitted in the Peter Bowman Jr. moniker: while Exhausting Moment focuses on the acousmatisation of sound’s source, the An Arrangement music video articulates long envelope synch-points to languid notes that are slow to reach their maximum volume. Exhausting Moment was developed incrementally over a period of three years, which presents an example in which the songwriting process continued past the audio recording and into the stages of creating the video. At the level of synchresis in production, the video illustrates an interplay between various acousmatised sound sources, or sounds that are produced off-screen by a source that is never depicted visually. In an effort to display the recording or playback media that is being used throughout the video, the visuals are comprised from shots of the video process itself: of the editing timeline, cameras re- photographing aspects of the video editing process.
Pap—Self-Titled (2019)
The songs recorded in the Pap moniker are a collaborative result of improvised recordings, which have been revised, cut, and structured into songs: a process of “of discovery; of making marks, deleting them, remaking them, adjusting them and tweaking them until we arrive at the desired result” (Owen, 2011, p. 316). Rather than recording using a moniker, Pap as a collaboration presents a unique approach in the submission, as the songs were written by recording and revising accidental marks. To the artefact, Pap signifies a different method for the recording of music, as improvisation and error is used throughout the process. In contrast to the other monikers, the songs are written in a variety of musical styles that are significant for their being comprised of improvised recordings.
The broad aim of the Pap recording project has been to explore an ambitious sound palette, and means for generating recordings, which are less constrained by intervention. Rather than attempt to begin with a pre-determined idea of the song’s aesthetic and how medium define a more evocative aesthetic, the songs in the Pap moniker are created by using long periods of improvisation and revision. In doing so, the songs are constructed with long passages of improvisation – often for hours, until each song is cut down to a few minutes. More so than the monikers in the submission, Pap strives for an ambitious starting point for which to create synchresis. The eleven songs and one music video submitted are a collaboration between my friend and colleague Matthew Conrad and I and as such are the only authentic collaborations in the submission of artefacts. Via their status as improvised recordings, the song ideas are from the collaboration, by interchanging a microphone between each other.
50
Omitted Compilation
The excerpts of the omitted compilation are comparatively short portions of the unfinished synchretic experiments in the artefact submission. The descriptions and reflections refer to incomplete videos, or works in progress, that are linked in a compilation video rather than submitted as a disparate range of short video files. The inclusion of these excepts, while incomplete, are intended as supporting material, to illustrate approaches to the creation of a synch-point which are not evident elsewhere in the project. As these examples were necessarily abandoned due to the overwhelming complexity of completing the entire project, their inclusion at times illustrate a more comprehensive illustration of the music than in the main submission. These omitted excerpts include Taking Over by Pap, in which the drips of paint are synchronised to the envelope of a melody. Scalar correspondences are illustrated in the excerpt of the Gary War music video for Thousand Yard Stare.
51
A Statement Addressing the Authenticity of the Album Covers in the Artefact Submission
To give the impression of the musical submission being produced in a physical format (released as a compact disc, cassette, vinyl record), each album has an accompanying album art. The various images of album art imply a distribution process, of each album having been transferred to a physical format. While this is loosely connected to synchresis, the album art creates a visual identity for each project, to delineate between each moniker. The images falsely give the impression of physical media, although they have been created in Adobe Photoshop as layered images, and can more accurately be considered photographic compositions, than as evidence of a photographic referent of an object.
Each artist is played by me in a different outfit, with the exception of the Len and The Dippers album cover which features Jaime Cartwright. The jewel-cases and stickers that appear on each album have been digitally manipulated to suit each cover, which have been placed using photoshop onto each distribution media. The aim of creating an accompanying album cover for each moniker is briefly discussed in this section as an extra step in the submission, to delineate between the style of each moniker with an identity that supports the music video and songs in the artefact submission. While the adjoining album covers are not necessary for a study of synchresis, the implication of the album covers are used to suggest the style and genres contained in each.
The inclusion of the album art is relevant to the key focus of the thesis, as the images provide an unconscious and crucial way of engaging with the music. The album covers reflect what is to be expected by the music, as Lyndon and Simon observe part of music’s power is in multimodal communication: that through the lyrics, images and musical sound, posters, films and album covers are used “to enhance and make specific, music’s semiotic meanings” (Lyndon & Simon, 2017, p. 8). The album covers pose a question asked by all covers: “what does the sleeve art tell me about the musical journey I’m about to take?” (Black, 2016, p. 17).
The process of re-photography, or photographing another photograph, was used in the creation of the album covers to undermine their artifactuality and to make the images of hard-copy formats seem more authentic as physical objects from the noise and obscurity that arises in the process of creating a duplicate. Debates around the process of re-photography are inherent to the idea of deliberately undermining “the notion of artifactuality”, as an artefact need not be a physical object”, which is further defined as “anything man-made; a product of human art or craftsmanship” (Guter, 2010, p. 9).
52
Bond suggests that while “noise” and “audible hiss” are an intrinsic aspect of audio recording using magnetic tape, the “photocopy” or “dupe photograph” reflects a similar process, stating “the copying process is often recognisable in a narrowing of the original colour gamut that is manifested in a loss of detail in shadow areas” (Bond, 2012, p. 20). As reflected by generational loss of audio and analogue video recordings, the re-photography of the album cover images also has the effect of adding entropy with each generation. This process, colloquially referred to as “making dupes”: the process of making a generation or analogue copy, means the newer version will “always marked by the process: a consequence of making the copy is that the new image is degraded” (Bond, 2012).
Figure 2: Re-photographed album covers depicting the recording monikers.
The duplication of an analogue recording or each consecutive “generation” introduces entropy, which Bond characterises as the “inherent degradation” of analogue reproduction, in contrast with a digital process in which information is duplicated without degrading the quality of the recording (Bond, 2012, p. 20). The use of re-photography is to introduce analogue noise and distortion to an image, to imbed the images with a sense of false realism and obscurity. Additionally, the album covers imply the music as having been distributed as physical objects, though each cover is created through a process of digitising degraded images and layering the images were layered together in Adobe Photoshop. The
53
photographic manipulation, in combination with medium-specific noise, contributes to the misinformation of a false archive. Medium-specific noise is used to obscure the musical recordings and suggest the songs are authentically created in a previous era, while visual noise is used in the production of album cover images to imply the authenticity of the recording format as a real object. The final image in Adobe Photoshop would be re-photographed from a computer monitor using a VHS camera.
In the following section, using Chion’s discussion and terminology, I have suggested a taxonomy for instances of audio-visual synchronism as hierarchical relationships. This is to suggest synchronism occurs as synch-points, or single-instances of synchrony between the audible and visual modes are spotted throughout a moving timeline. This occurs as the presciently created mode forms a synchretic envelope which is transduced into the shadowing mode: as either the audio information or video information which is created in advance, catalyses the production in the other which is created afterwards. The synchretic parameters in a synchretic envelope are broadly divided into accented or sustained fusions: collision and cutting in the short-envelopes, and dynamic and transmutation within long-envelopes. As the two key sites of production which are reflected in the submission of artefacts, commercial advertising and music video provide areas of production in which I have identified instances of synchretic cutting, collision, dynamism and transmutation.
54
Synchretic Cutting
Editing is described by Bordwell and Thompson as “the process of joining together two or more shots” that is rooted in temporal and spatial dynamics, and is intended to shape “the trajectories of movement across shots, scenes and sequences” (Bordwell & Thompson, 2012, p. 192). A cut is the most immediate editing gesture, enacting an “instantaneous change” between moving images, which in this section refers to being produced to create a synch-point (Dancyger, 2011, p. 437; Wallis, 2008, p. 18). In parallel with transitions requiring a sustained fusion, such as a wipe, dissolve or fade, a cut is classified as a short-envelope synch-point which is produced instantaneously, from one frame to the next.
Music videos are a fragmented form, in their reliance on collapsing of spatial and narrative cohesion with montage and jump-cutting (Donnelly, 2015, p. 171). Cutting in the context of a music video fulfils the dual purpose of synchronising to and visualising the rhythms in the music, while conveying the flow of a narrative (Vernallis, 2008, p. 125). The process of editing to music relies on kinesthetic empathy and corporeal imagination, or on innately bodily experiences to judge read the rhythm of images, similarly to a musicians body participating in transmitting musical rhythm (Frierson, 2017, p. 336). This occurs at the stage of refining a sequence in post-production, that involves the winnowing and intuitive search for “clarity and dynamism” during which the sequence of images are shaped by the music, which place an importance on the jump-cut rather than the match-cut (Dancyger, 2011).
Certain kinds of synch-points are deployed so frequently they are emblematic of a music video, for instance a jump-cut to the song’s rhythm to interrupt the flow of images in synchronization by instantaneously cutting to a different moment in the same shot (Donnelly, 2015, p. 171). A synch-point adds rhythm at the placement of a cut in synchronisation with the starting sound, or the “attack transient” of the envelope of a note or beat. For instance, in the music video for the Jay-Z song Dirt Off Your Shoulder (Meyers, 2004), jump-cutting to the beat is used during a shot of a car approaching, as the camera pans to the front-wheel. The car’s approach is framed from a low angle, as jump-cuts fragment the slow-motion exit of the people from the vehicle in time with the song’s kick drum. The camera movement and the elements of the scene are not choreographed to respond to the music, other than via the cutting, which has been added at the editing stage. With a lack of responding motion in the mise-en-scene, cutting in synch exemplifies rhythm added in post-production by fragmenting the sequence in alignment with rhythms of a song.
55
Figure 3: Jump-cutting in the music video for Dirt Off Your Shoulder.
In the Michel Gondry music video for The Hardest Button to Button (Gondry, 2001), synch-points are forged between the beat of the kick drum and timing of jump-cuts. The music video depicts the two band members, each accompanied by a guitar, an amplifier and drumkit, playing in various locations, such as: in parks, on roads, under bridges, in train stations and open fields. With each kick drum and bass note accenting the pulse of 4/4 beat, a cut occurs and the band is teleported further along the path with a new amplifier and drumkit, leaving a trail of instruments behind them. Each cut of two band members moving to the next iteration of musical instruments is reflective of the starkness of the musical accent, which transposes the band to the next location. The cut is immediate in its transposition of the band members: teleporting them from one frame to the next, while continuing the line of instruments. Likewise, the Jay-Z example uses jump cutting to accent rhythm, though in The White Stripes music video the exploration of cutting is intrinsic to the sound, and more deeply investigated in articulating patterns and the staccato rhythm of the song, rather than being deployed solely as choreography.
Figure 4: Cutting to the beat in The Hardest Button to Button.
In the introduction and bridge sections of the music video for Enjoy the Silence (Corbijn, 2005) by Depeche Mode, synch-points are forged as painted flowers are aligned with synthesiser chords on the first beat of each bar. This is also demonstrated in the music video for The Show Must Go On by Queen (QueenOfficial, 2013, Oct 16). In Enjoy the Silence, the cut visually divides the black and white shots of the musicians with flash-frames of colour to the percussive accents in the song. The beat has an incremental effect which builds on the sequence, as with each repetition of the four bar bridge, a band member is removed until the singer remains. In cutting to a contrasting scene, the visual accent reinforces the musical repetitions, while the elimination of band members has a wider effect of 56
incrementally fulfilling the sequence. This is similarly seen by intercutting a shot with glimpses of another scene to music can forge a synch-point by interrupting the succession of moving images with a visual contrast.
Intercutting in synchronisation with a musical idea is used in the second verse of the Beastie Boys music video for Intergalactic (BeastieBoys, 2009, 15 Sep), as each time a vocal track is doubled at the end of a phrase, a cut occurs and the band framed similarly in another setting performing the same lyric. Certain words enunciated by the layering of vocal tracks are visually reinforced by cutting to the accent, which rather than add rhythm, provide a brief interruption to the flow of images. In the video for Hey Baby (NoDoubtTV, 2009, Oct 7), as the camera zooms in on two band members the image strobes a black screen in time with the laser sounds in the music. As the sound of a laser repeats at a slower rate and lowers in pitch, the strobes with blacks slow as the two band members near each other as the camera zooms in. Similarly, in the music video for Gold Digger (KanyeWest, 2009, 17 Jun), the screen rapidly intercuts with the sound of sound of rapid kick drums. Rather mickey-mousing in which the morphology of sound and image are traced in unison, the shots are cut to the rhythm of the song and dispersed in a manner that is one of many audio-visual tropes, than as the basis for the video itself.
Figure 5: Cutting to black in the introduction of Hey Baby.
The music video for the 1984 pop song by Aha Take On Me (a-ha, 2010, Jan 7) revolves around a comic book character leaving the boundaries of a comic book, growing to human proportions and interacting with the protagonist. The narrative begins as a young woman reads from a comic book, depicted in the music video by extreme close-ups of individual frames, only later establishing the setting of the protagonist’s bedroom. During the introductory sequence, cutting is used to align synch- points with the rhythms in the song, as on the first and third beats of the bar in the introduction, close- ups of comic book panels are cut in time with kick drum hits. The synch-point is not created by motion within the mise-en-scene or camera movement, but from the cut to the next still frame. The cutting is used to reflect the stillness of the frames of the comic, by restricting the syntax of the shots to still close-ups.
A recurring type of cutting in synch occurs with a camera shutter sound and freeze-frame of the sequence. The use of freeze-framing, or abruptly pausing the playback of a shot in time with the music. Like the jump cut, a synch-point aligned at the moment a sequence pauses can be established 57
despite the shot having no previously intended rhythmic characteristics. For instance, in the music video for Baby Birkin (Gunna, 2019, Jul 9), the sound of a camera shutter is used on top of the mix, as a rapid sequence of still-photos are shown in synch to a camera shutter sound effects added to the music. Other examples show a variation of this technique, as the section of a song comes to a close, the performers holding a camera and taking a selfie freeze-frame in synch with the end of the bar (BoyMeetsGirlVEVO, 2014, Feb 21).
Figure 6: Cutting to the beat in Take On Me.
Other instantiations of the technique depict the synching of a sound effect with a sequence of freeze- framed images (Drake, 2020, Aug 13). The snapping of the camera effect implies an instant memory that in a sense has a canonizing effect on the audio-visual information. The freezing of the sequence combined with a shutter sound effect which implies a photo being taken, and a moment to be celebrated that is worthy of capturing a photo. In the Gunna music video this is a rapid sequence showing money in a bag, or in the Drake (Drake, 2020, Aug 13) music video standing against a photo studio backdrop with the performers having their photo taken.
Particular examples illustrate an approach to cutting in synch which reflects the looping of a sample, which forms a relationship to the repetition of a shot. In the Oneohtrix Point Never music video for Sleep Dealer (software, 2011, Oct 27), the music video shows a person pouring two cans of soft drink into empty glasses placed either side of them. The video is split into two sections according to the structure of the song, one that is synchronous and one that is asynchronous. The A section of the song depicts synchresis from an audio sample of a sound that is suggestive of the satisfaction of a drinking a carbonated drink, that is looping and repeating in short succession. The song is built around a repeating audio sample, that with each loop the video is repeated in synch, until across the various sections of the song the glass is incrementally filled. The linking to a visual source is reminiscent of Warner’s (2017) discussion of extreme congruence, of footage and a sound being recorded and played back simultaneously in a musical structure, which necessitates the playback of a sound accompanied by the visual source at the moment of recording. Contrastingly, though the audio and video were not recorded simultaneously, the use of the sample being repeated to the cutting of a video loop, gives the impression of their inextricability of sync-sound recordings.
58
Peter Bowman Jr.: Exhausting Moment
In drawing from Schaeffer’s terminology introduced earlier in this text, Chion suggests acousmatised sound is neither inside nor outside the image: neither inside as the visual emitter of sound is visually unrepresented, nor outside as the sound is still implicated in the action (Chion, 1994, p. 129). The acousmetre is disembodied from its source, often in the form of a narrator holding power over the scene: “seeing all, omniscience, omnipotence, ubiquity” (Chion, 1994, p. 130). This power, Chion (1994) suggests, is lost as the voice is de-acousmatised, or as the visual source of a voice is revealed, a process which Chion refers to as embodiment (p. 131). The neologism acousmatic is derived from “the ancient Greek word akousmatikoi, the name given to the disciples of Pythagoras who listened to the master’s lectures through a curtain”, according to the legend Pythagoras’ body “was hidden from the akousmatikoi, leaving them with only the sound of their master’s voice” (Kane, 2014, p. 24). In returning to the Schaeffer’s definition introduced earlier in this thesis, acousmatised sounds refer to “a noise that is heard without the causes from which it comes being seen” (Schaeffer, 1966, p. 63 and 64).
In Exhausting Moment, acousmatised sounds imply the music video is being edited by a person off- screen, audibly operating the playback of the video technology which is transmitting the images. As well as being synchronised to the music, synch-points are aligned to the background room atmos, to the sounds of buttons being pressed and the mechanisms of a video player. Although the audio of room hiss and the screeching of the tape mechanisms are unmusical, their starkness suggest an unedited audio-track of a space offscreen uncannily in time with the rhythms of the song. Across the course of the video synchronisation occurs in time with the music and by way of interruption: between captured moments of video distortion and erroneous gesture synchronised with the room sounds.
The room sounds are acousmatised or deprived of their visual counterpart, to allow for speculation “upon or inferring causal sources”, as is illustrated by sources of off-screen sound combined with videotape errors to form synch-points (Kane, 2014, p. 24). The cutting is also forged where there is no instrumental accent in the song, allowing both modes to dictate the other simultaneously. This approach to synchronization is not typical of the production of a music video, as the song cannot be altered once it is complete. In other monikers and music videos in the submission of artefacts, such as, in The Fed-Up Kids: The Heel, the melodies and rhythms catalyse the placement of a synch-point along the timeline and the length of its envelope. In Exhausting Moment, the various audio and video tracks interchangeably take the role of the leading mode to which the others align; the clicking and screeching sounds of the tape being operated are aligned with the sounds of breathing and furniture moving in the room, which may also catalyse the placement of a melody or rhythm in the song.
59
Cutting emerges from the recurring motif of depicting stages of the video timeline being edited in Adobe Premiere Pro. The aim of this is to conflate the accompanying medium-specific noise which is typically produced by a particular technology, by pairing them to the sounds of another technology, for instance: mouse clicks with VHS noise; the sound of a VCR being operated with images of a website; the rewinding of VHS tape to reveal the source of playback on a mobile phone. While the acousmatised sounds are never visually revealed, a space is implied during instances in which the screen turns black: revealing the reflection and glare from the glass of the television in the room. This is to imply the space in which the screen displaying the video is situated, by dimly revealing the area in the reflection during moments when the screen has no playback occurring.
The recordings having the status of discrete digital content in Adobe Premiere Pro facilitates the combination of analogue video recordings in tandem with low-definition digital and high-definition digital video. Once the footage had been captured in a discrete digital form; it was sequenced in a non- linear editing timeline and then re-recorded onto other formats: as screen recordings, mobile phone images and videotape. For example, the editing timeline is depicted in Adobe Premiere Pro intermittently pauses and reduces in size to reveal the image is a recording of the program that is being played back in a media player. The significance was to undermine through a synchretic experiment the kinds of accompanying expressive noise and the pairing of a visual and audio media that a viewer would expect from recording formats.
Figure 7: A digital glass breaking effect on videotape footage in Exhausting Moment.
60
Synchretic Convergence
In commercial advertising there are examples of a coda to a commercial occurring in synchronisation, by pairing a musical accent with a cut to a tableau of the advertised items. This moment of fusion reflects the apotheosis of the video sequence, which typically re-enters following a brief silence with the tableau of the product to punctuate the timeline. The resolution of the sequence as a still arrangement of the product in synchronism with the penultimate moment of a musical phrase is reminiscent of Cook’s (1992) discussion of convergence. Cook states the separate “tributaries” of language, writing and speech converge to articulate “a single message” in the final moments of the sequence (Cook, 1992, p. 38). This parallels Chion’s definition of a synch-point of convergence as “a form of punctuation at the end of a sequence whose tracks seemed separate until they end up together” (Chion, 1994, p. 59).
The synchretic resolution using a tableau is invoked in commercial advertising for Victoria Bitter (oztvheritage, 2012, Oct 12), which typically depicts a montage of people in working class jobs and activities guided by a voice over. As the narrator states, “you can get it jumping,” a shot of a cowboy riding a bull is shown, or when “you can get it pressing a suit” is said, a man steaming a shirt gazes into the distance. This culminates with the narrator changing the pattern of the montage, as the music halts momentarily he states, “as a matter of fact, I’ve got it now”. The music re-enters and the instruments reinforce a final accent, cutting in synch with the musical accent to a still-life of Victoria Bitter filled glasses, bottles and cans beaded with perspiration.
The commercials are typically structured as a montage depicting Australians in a myriad of working activities which culminate in a tableau of a staged vignette of Victoria Bitter bottles and cans of beer. The montage depicts physical labour interspersed with shots of people fantasising during the workday, the connotation of the resolution to their fantasy being the resolution to the advertisement. This penultimately resolves in an audio-visual synch-point at which the music stops, suggesting the beer is the reward for their labour by the sequencing the resolve of the music and of the video sequence with the beer. As the montage of pensive individuals culminates with the beer, the implication is what is being fantasised about is the ideal reward for their labour. This is further connoted by the unification of the beer at the end of the ad, as an anthropomorphised family of beer, presented as a group of individuals within a unified group.
61
Figure 8: A sequence of images in a Victoria Bitter television commercial.
Verbal Signification and Synchresis
Chion (1994) observes a synch-point may be forged from spoken dialogue, that by “its affective or semantic character” can convey meaning and become locus of synchronisation (Chion, 1994, p. 59). This occurs as a visual cut is coordinated to emphasise the voiceover, or as alternatively, a sign that represents the spoken word may occur, “the singer may sing the word ’purse,’ and the viewer will see the image of a purse” (Vernallis, 2004, p. 182). This is depicted in an advertisement for Coast soap (513official4, 2011, 19 Mar), as the protagonist opens and uses a bar of Coast soap in the shower, suggests a trip to the park with his dog, then runs from his house and freeze-frames in a celebratory jump in the air.
Figure 9: Verbal signification in a Coast soap commercial.
The word Coast is synchronised with the image of the soap, which is reinforced by its repetition. In this gesture, the signifier and signified are presented at once: as the bottle is shown on screen and the singer exclaims “Coast!”. This is exemplified as an extreme close-up is framed, portraying the man smelling the soap and smiling, the shot tilting up to his eyes as the singer states: “the scent opens your eyes”. The man soaps his arms the singer continues, “Coast, lather! Spirits start to rise!”, at which point the visuals respond as the song increases in intensity. This is repeated as water strikes the soap from behind while the man jumps from his porch with his dog, synchronising his exaltation with the lyrics. Throughout the structure of the commercial, with each repetition of the signifier there are consonant visual images accompanying the product name with representations of positive emotion.
Examples of verbal signification occur between a spoken word and an image of the word animated into the composition. In a 1992 clothes detergent commercial for All (gamelovercommercials, 2013, May 62
16), as the letters of the brand are sung in the song, a hand comes into the screen and synchronises to fuse a synch-point between the finger touching each letter on the box. The example displays triangulation between the music, the lyrics and the brand name with a synch-point. Throughout the advertisement, which could be characterized as asynchronous, there is some relationship between the lyrics and the progression of the sequence: as dirty water from off screen hits a child, the lyrics over the top are “collects so much dirt”, while later as a sauce stain hits a shirt “oily icky stains” is sung.
Figure 10: Synch-points in an All commercial.
The moment at which synchronisation occurs is only once in the structure of the song, during which each instrument collectively reinforces the musical accent, which is the resolution to the previously shown glimpses of collecting dirt. While there is only one synch-point throughout the sequence, which occurs as the letters on the box are shown, the synch-point conflates certain elements which have been introduced and distributed throughout the course of the commercial. The lyrics supply information about the product, although this is musical and part of the vocals to a song, rather than a voiceover. Furthermore, this is shared by the singing voices of an old man and a child, although as the synch- point occurs, the letters are sung by the older man as the child’s hand touches the box.
It’s The Delmer Brothers!: Cindy Doll and L’Efforel
Verbal and linguistic cues of syncretic convergence are present in the two music videos of the It’s The Delmer Brothers! moniker. These two works are created in the style of failed vintage 1980s television commercials, one for a perfume and one for a toy doll, to explore the emergence of synch-points by the “musical, visual, and verbal” cues at play in advertising, particularly in the way the voiceover produces a moving image (Scott, 1990). As sound is the leading mode, the music was recorded presciently to the moving images to which they are synchronised, by recording onto a multitrack cassette machine and arranging the layers in Ableton Live. Instruments
63
In the video, the narrator’s speech is visualized by text and graphic animation to accent the voiceover and lyrics of the advertising message. As the lyrics follow the melody: “life is so great with my Cindy Doll, she’s my new best friend,” each word appears inside a star-shaped graphic. These synchronous animations are intended as self-reflexive gestures, which in their simplicity, ghostly glow and desaturated image, create a perverse inversion of Almeida’s description of the toy commercial characteristically using verbal and visual cues “which intermingle to convey significations that encourage a relation of friendship” (Almeida, 2009, p. 496). The song vocals are turned down to give prominence to the narration, as Chion observes the voice is isolated in the sound mix, to which “other sounds (music and noise) are merely the accompaniment” (Chion, 1994, p. 6).
Figure 11: Videotaped animation in the music video for Cindy Doll.
The image of the doll was re-photographed to pursue the visual style of a vintage commercial with noise and distortion to match the quality of the cassette recording. Rocking motions and lip-synching were animated to convey the toy’s speech malfunctioning, which appear via speech-bubbles in synch with the vocal samples. The synch-points occur via simple animation that is characteristic of vintage toy advertising: as lights blink to the snare and kick drum strikes, the doll tilts rhythmically to the melody and lip-synchs phrases of the lyrics and narration. To impart a sinister connotation through the degradation of the image and sound quality, the colours are streaked and glowing from multiple analogue generations. This process of degrading the images is to match the audio quality of the tape, with noisy and distorted video and audio-tracks in tandem that pervert and obscure the intent of the advertisement. Like the accompanying song for the L’Efforel music video, the recording process using a cassette and Ableton Live imposed linear restrictions on the process, in which the individual instrumental or vocal tracks were not mutable once the noise and distortion had been added.
The adjoining video for It’s the Delmer Brothers: L’Efforel is produced in the style of a vintage cologne commercial, which similarly explores a triangulation between the voiceover, sound effects and music as the basis for the occurrence of a synch-point. As the sound recordings are comprised of music and voiceover narration that have been captured presciently to the video, the audio catalyses the emergence of signs that are produced visually. The production of objects in the scene of the video was achieved using the open-source Blender software for the purposes of animating the object models, their movement, texture, and the environment in which are set. The 3D models in the music video 64
were creative common files and stock objects from within the Blender software, which were used as a basis for a mutable environment, in which the alignment of parameters to the song could be revised without degrading the image. While the objects (wine glasses, blankets, shadows) in the scene are computer-generated, the addition of visual noise and distortion produced from videotape generations is used to imply a vintage commercial.
Similarly to the Cindy Doll video, the synch-points are triangulated between the voiceover, music and sound effects, by text animated into the mise-en-scene to verbal cues in the voiceover. Cursive text emerges in synchronization with the spoken narration in the soundtrack: “for the man who wants it all… L’Efforel. When war is won, but never lost… L’Efforel.” As is emblematic of vintage commercials, each time the product name is repeated a perfume bottle emerges, it is sung by a chorus stating the products name in a seductive tone “L’Efforel”. As the shadows of falling roses, diaphanous fabric, wine glasses, cursive text, the ocean, bedsheets and wine glasses fall through the frame, past stark chiaroscuro shadows and angular beams of light, they are cued to percussion and vocal cues.
65
Synchretic Collision
Collision synch-points emerge from surface contact in the mise-en-scene, which are forged between a visual of two objects touching with the attack transient of a sound’s envelope, or the “starting noises” of a sound (Avanzini, Rath, & Rocchesso, 2002, p. 445). Synchronisation that occurs via surface collision suggests the texture, mass and material of an object and/or of an environment, which reflects Chion’s discussion of surface and spatial characteristics or the materialising indices of a sound. The materialising indices, he states, give information about the “material conditions” of the sound, and refer to the “the concrete process of the sound's production” (Chion, 1994, p. 114). As this category is characterised by fusing sounds to surfaces touching, the envelope may be in part short and long envelope: short as in momentary contact as when ball hits a racket, or as an instance of an impact depicts continuous contact of a long-envelope, such as friction, rolling, or glass breaking. The importance of collision is the musical context of this project, from which it emerges as an important feature to underscore the timbre and texture of instrumentation in a song.
A synch-point of collision supplies information about the material and weight of simulated and live- action objects, which is ubiquitous with the task of foley artists and the animation of hard and soft-body objects in the realistic auralization of virtual environments (Ren, Yeh, & Lin, 2010, p. 139). As Chion suggests in his discussion of explosions, whether they are animations or created with substitutes of cement or glass using non-resistant materials, they gain their “consistency and materiality” through sound (Chion, 1994, p. 5). The surface collision of an object and its accompanying sound provides information about the object and the environment, the sound design simulates the contact of virtual objects “to elicit perception of material” (Avanzini et al., 2002, p. 445).
The examples in this section are illustrative of musical cues, which are necessary to delineate from the underscoring of naturalistic action. As such, this discussion leaves by the wayside instances of foley sound, such as footsteps and doors knocking, with the intention to focus on examples of collision that are visually expressive of a musical idea. To focus on instances of material similarity in a synch-point, this section begins with examples of collision in toy advertising that are paired between motion and sound effects, followed by examples of surface material being synchronised with musical cues in music video. The discussion of material is of the artefact submission The Winnings and Losses of Dr Kyotan, in which the material of surfaces is reflected in moments of collision.
66
Synchronism produced via Materialising Indices
Sound effects are used in the presentation of toy commercials, that are typically spotted to enhance the action or function of a toy, for instance as a projectile is fired, a door opens/shuts, or vehicles collide. Certain examples also demonstrate synchronization of a toy’s movement with sound effects and musical cues, which are imitative of the genre tropes of action films (Starr, 1985). As marketing strategies are oriented around children’s “imaginative play”, advertising and its producers have a role in defining “the limits of children’s imaginations” (Greenfield et al., 1990, p. 238). As Greenfield suggests previous research (Greenfield, Farrar, & Beagles-Roos, 1986; Kunkel, 1984; Meline, 1976) indicates that contrastingly to print or audiotape, the audio-visual medium may inhibit creative imagination. Rather than reflect personal experience, these associations may be restrictive to the imagination, as the toys act as visual cues to recall the narratives and action of an associated program (Greenfield et al., 1990, p. 239).
Figure 12: A toy projectile being fired in synchronization with a sound effect.
In surveying vintage figurine commercials of the 1960s-1980s, certain uses demonstrate uses of audio-visual synchronism that underscore the motion of a toy. Collision synch-points are spotted to augment, or “puff” a toy with sound-effects and music which are not included in the advertised item fused to realistic or “toylike” sound effects, or with musical cues. In commercials for GI Joe (8thManDVD.com™CartoonChannel, 2014, Mar 9), the militaristic aesthetic is reflected in the synchronised battle and war sound effects, along with musical cues which loosely dramatize the narrative. As a toy bomb deploys from a military raft, a laser sound accents the projectile. Similarly, a commercial for Terminator 2 figurines shows a miniature wall being destroyed, which is fused to sound effects of screeching tires and crashing as the toy car screeches and impacts with a building (ЮмарвинаСмешинка100%, 2019, 12 May). The accents which are synchronised have a cinematic quality from their pairing to sound effects that would not be present in the actual use of the toy, which underscore the moment of collision while exceeding the imaginative play that the toy has as part of its function. 67
Figure 13: A projectile being deployed with a sound effect.
In contrast to being synchronised with recorded sounds of explosions and bricks crumbling, collision synch-points are reinforced through sound effects which are made verbally by children. Synch-points are forged to the vocally produced imitations of battle sounds in the commercial, which are synchronised with camera motion in the mise-en-scene. In other examples the morphology of synchronisation is triangulated between sound effects, the music and the visuals, for instance as a toy lands on a surface as in a commercial for the boardgame Mousetrap (LerchVid, 2018, Mar 18). As a toy descends a slide and lands in a container, a downward whistle and a cymbal crash accompany its gesture and the moment of collision. As synch-points are spotted throughout to depict playing, the synchronised sound effects and musical cues are intertwined with the sound included in the toy’s function. The synchronism conflates the audio-visual fantasy with the real product, depicting a marketed fantasy that has “become highly ritualised” in using cinematic tropes to define the limits of the imagination (Jhally, 2002, p. 332).
Figure 14: Collision synch-points with sound effects in a toy commercial.
In Chion’s (1994) discussion, the materialising indices draw the scene “toward the material and concrete” or in sparse occurrence can draw the story and characters towards the “ethereal, abstract and fluid” (Chion, 1994, p. 114). Chion (1994) suggests they cause the listener “to "feel" the material conditions of the sound source (Chion, 1994, p. 114). In the music video for the Ian Van Dahl song Castles in the Sky (ТехноМикс90, 2016, Aug 2), synch-points are formed to musical accents in an otherwise a-synchronous sequence, or in Chion’s (1994) characterisation are constrained to intermittent “synchronization here and there to solder the music and image together” (Chion, 1994, p. 82). As reserved footage of a glass sphere exploding to the sound of a reversed cymbal, the liquid
68
inside is sent into orbit in synchronization with a rising crescendo. As the transparent glass sphere smashes, there are two parameters at play in the synchretic envelope: between the reversal of the sound in correspondence with the reversal of a shot, and between the texture of the object. This articulates the materialising indices that granularly dissipate as the image similarly responds with the dissipation of an object into fragments.
Figure 15: Glass breaking in the music video for Castles in the Sky.
The materialising indices of a sound and image are exemplified in the music video for Crawling (LinkinPark, 2009, Oct 24), that begins with the sound of a murmuring wind to close-up of a water filled sink, the silence and stillness of which are broken in-sync by a high-pitched melody and droplets of blood against the water. The collision of the droplets of blood reflect the attack transients of the synthesiser melody, conveying a link between the materialising indices of the high-pitch, glass-like timbre implied by each droplet. The gesture of collision fuses the lightness and texture of the sound to the spatial and textural aspects of the droplets striking the surface of water. In reflection of the lightness, pitch and timbre, the image in synchronization catalyses the beginning of the song by breaking the stillness of the image.
Figure 16: A droplet hitting water in the Crawling music video.
With improvements in resolutions, recording formats, and the increasing power of editing workstations, music videos more recently bear a multitude of transparent video layers and overlays. As one of three main techniques of generating synthetic motion, keyframing or “manually specifying motion” allows for careful manipulation of layers (O'Brien & Hodgins, 2000, p. 69). Increasingly synchresis emerges from the combination of keyframing live action footage, filters and transparent overlays aligned to music and sound effects. In a video by Gunna feat. Young Thug (Gunna, 2019, May 7) the camera throughout zooms in slightly with each kickdrum sound, a variation of which is seen in a music video for 69
Moneybagg Yo (MoneyBaggYo, 2020, Jul 17), and in the Future music video for Posted With Demons (Future, 2020, Jul 20). In each example short-envelope synch-points are forged between the kick drum and the zoom function, or keyframing the scale or focal length of a 3D camera.
A combination of zooming in and out to the beat of the song is used in the music video for You Need To Calm Down (TaylorSwift, 2019, Jun 17). Similarly, prior to affording the enlargement of an image without degrading the resolution, the same technique is achieved by varying the shot type in synchronisation with a drum fill in the music video for Take Me Home (PhilCollins, 2010, May 11). In each of the examples the “punching-in” effect creates short-envelope synch-points, which are accents too brief to depict any trace of melody or morphology. Contrastingly to the pre-prepared and scored synchronisation during the early stages of animation history, the synch-points are afforded freedom of applying a keyframe at any point, or of manipulating any parameter of the audio-visual at any point along a sequence. Rhythm can be added when deemed appropriate: with a similarity to cutting in synch, in which the scale can be enlarged without losing image quality.
Figure 17: Zooming in and out to create a synch point in You Need to Calm Down.
The ability to complexify the arrangement of layers in the composition of visual elements means synchresis can emerge by augmenting keyframes at any point in post-production, and by doing so relate the lyrics in synchronisation with a performer’s actions. In the music video for Butterfly Effect (TravisScott, 2017, July 13) a range of media formats, layers with varying opacities and blend modes work together to align to a synch-point. Fusions emerge disparately in the sequence and are responsive to lyrics, as for instance a performer points their fingers like a gun in synchronisation a vocalized “bang”, a gunshot sound effect muzzle flash is animated to the end of the performer’s hand. As with the previous examples of zooming to create a synch-point, the example shows the augmentation of live-action footage using transparent layers, that may afford instances of improvisation which can be augmented in post-production.
In a music video by Asap Ferg (Ferg, 2020, Sep 25), during a basketball game sequence, cutaways show the ball being thrown through the hoop. The sequence of the basketball entering the hoop is synchronised to an interjecting backing vocal, or a vocal ad-lib imitating a “woosh” sound. The imitation of wind that is vocally produced might be characterised as an ideophone, or a word that invokes an
70
idea in sound: the ideophone in this case being an imitation of the sound produced of a basketball going through the net, which is synchronised to the image of the ball going through the hoop.
A similar matching occurs in the music video for Jobs (CityGirls, 2020, Jul 13), as a woman applies lipsticks in front of a mirror and “smacks” her lips in synchronisation to the handclap of the beat. These connections are often established more loosely, for instance in a video for Blueface (bluefacebleedem, 2020, Jul 9) as the lyrics state “we leaving them fried” flames are composited instantly into the shot. Such examples and uses of the technology in this way suggest a reliance on the technology itself, in which the ability to align audio-visual parameters in post-production is relied on more than the careful choreography of motion to sound in pre-production.
Figure 18: Using filters in synchronisation with lyrics to create a synch-point.
Pap: The Winnings and Losses of Dr Kyotan
The omitted excerpt of the video for The Winnings and Losses of Dr Kyotan explores collision between sound effects, music and computer-generated objects (CGI). In exploring transmutation, the synch- points are animated to change their surface texture on impact, shifting between rigid and soft body objects during instances of collision. Similarly to the examples in toy commercials, this experiment triangulates synchronisation between the images, sound effects and music. The synch-points in the video articulate the materialising indices of an object’s surface and the implied size of the space in which the collision is occurring. As Chion suggests, these supply “information about the substance causing the sound-wood, metal, paper, cloth”, that cause the listener “to "feel" the material conditions of the sound source (Chion, 1994, p. 114). As for instance a metal cube falls out of frame when colliding with an object, its surface and mass changes from metal to liquid in timed to the song’s rhythms, while also being reflective of the sound of the material.
In the conception of the video, there was no intended symbolic or thematic relationship in the choice of 3D models, beyond their supplying a variation of object size, texture and weight. With a focus on illustrating the materialising indices, the video depicts a continuous shot of various objects falling past the camera and receding into the distance until they collide with an invisible wall and move off-screen. 71
The surface texture and mass of each object are implied as an object strikes the invisible back-wall, with a correlating sound to match the acoustic, timbral or textural aspects of the object.
Figure 19: An object’s texture and material changing in The Winnings and Losses of Dr Kyotan.
The moments of collision initiate a transmutation, as the synchronization reflects the changes of the texture of the sound and surface of an object together as they change from within. The use of the Blender 3D software allowed a wireframe object to be initially modelled, then wrapped in a range of materials and textures, to give the object the likeness of crystal or glass, sand, rubber, metal, fabric or mud. Using different materials, textures and weights, the process of creating computer generated models using Blender was ideal to depict objects morphing.
Each instrumental track in the song was recorded in Ableton Live and synchronised to the recurring motion of a 3D object. As for instance an orange collides with a wall it is synchronised with a snare drum, a sound to which each recurring impact of the orange is synchronised each time a collision occurs. As the synch-points illustrate spatial and material characteristics implied by the texture of the objects in a virtual environment, the texture and timbre of the instrumentation, in addition to the audio effects that are applied to an audio track are represented in synchronization.
Figure 20: The assignment of objects to instrumentation in The Winnings and Losses of Dr Kyotan.
72
An aspect of the materialising indices are “the effects of spatial acoustics”, or the “sensation of distance” between the source of a recording and a microphone, which by their reverb suggest the sound as having been produced in a specific space (Chion, 1994, p. 116). This is conveyed as the reverb applied to the collision sounds change to imitate the growing or shrinking room. As the scene is completely black with no visible objects or space, the growth of the space is implied by changing the position of the invisible floor with which the objects collide, leaving the object to fall for a longer distance before collision. To invoke a correlation between the spatial parameters of sound and video tracks, reverb is added to the percussion track to imply the room growing in scale, as reverb is introduced, the sound of the consistent rhythm track implies the space is expanding and ringing out as if they were gradually being projected from a cavernous space. As such, the music video is an attempt to articulate a complex relationship: between audible aspects of a scene that are implied via the sounds and the scale of the room that is not visualized.
Reinforcing a Short-Envelope Collision
Collision is used in music videos to underscore gestures, from the added value between the performer’s motion in the mise-en-scene to music and/or sound-effects. This is often effectively combined between aspects of the mise-en-scene synchronise with the choreography, as the environment and the performer’s surroundings respond with sound. For instance, weather becomes an actor in the augmentation of performances, as representations of natural settings through the invocation of “extreme-desert sands, deep tropical forests, oceans” become a performer’s malleable stage (Aufderheide, 1986, p. 70). In the music video for Thong Song (SisqoVEVO, 2009, July 17), a group of dancers perform on a beach, whose feet colliding with the ground is accentuated by a camera shake effect, in synchronization with a booming and rumbling sound.
Figure 21: A gap of silence and an accent in the music video for The Middle.
The sound effect to which the “stomp” is synchronised has been added to the song and is not an inherent aspect of the song’s musicality without the accompanying imagery. The sound acquires a relationship to the music via the imagery, as it is an aspect of the choreographed dance and supplies
73
an accent in time with the song’s metre.The synchronisation to sound effects that are external to the music is also used in the music video for The Middle by Jimmy Eat World (JimmyEatWorld, 2010, Jul 8), during the final chorus the band plays energetically in a house party of dancing people. The instrumentation is silenced for the first beat of the bar, then re-enters the mix as the image trembles, connoting the power and energy of the song. The synch-point is a singular occurrence as a brief gap in the song, and underlines re-entry of the instrumentation following the silence.
Multiple visual parameters aligning to reinforce a gesture of collision is illustrated in the music video for Betty Davis Eyes (KimCarnesVEVO, 2009, Feb 28) and similarly in the REM video for Shiny Happy People (remhq, 2009, Oct 26). During the chorus of Betty Davis Eyes, on the second and fourth beat of each bar, the dancers slap each other across the face in-synch with handclap sounds, with a fast attack transient, decay and release for the impact of each slap. The shape, or envelope of the clap sound is forged to the gesture, which reaches maximum amplitude rapidly. To reinforce the gesture, the actors being slapped swing their heads in the direction they are being struck and pause in a tableau and an overexposed blast of pale light is strobed on and off in time with the clap sounds.
Figure 22: Slapping used as collision in the music video for Betty Davis Eyes.
The Fed-Up Kids: School
The music video for the song School explores collision synch-points, which converge simultaneously from video layers of vehicles rewinding and fast-forwarding, exploding video game objects, and shaking flowers to instrumental cues. The song structures, rhythms and melodies of The Fed-Up Kids songs follow melodic phrases with little counterpoint, or “two, three or even more tunes playing at the same time” (Powell, 2011, p. 115). To convey the themes of naiveté, adolescence and rebellion inherent to The Fed-Up Kids moniker, synch-points converge from video layers of different recording formats, to align with and jointly reinforce the visualization of the same accent in the audio-track.
The process of assembling the sequence was to arrange a hierarchy of foreground and background video layers, to converge simultaneously with a musical gestalt. As such, different layers of video footage articulate the same synch-point together: high-definition digital video of flowers, VHS-C videotape footage, screen-captures of low-polygon game models and scrawled pencil-line animations. To combine multiple layers to reinforce the same musical cue, transparency was a necessary aspect 74
of displaying motion at different layers of depth reflecting the same musical accent, while other layers from their inability to be isolated would necessarily provide the background layer.
Figure 23: MIDI and guitar recordings in the Ableton Live session for School.
Synch-points are created in these background layers by tracking the footage (fast-forwarding, re- winding and pausing on a VCR) to provide rhythm underneath the motion of flowers shaking, pen scribbles and video game models exploding. The directional changes of the playback create synch- points to the rhythm and drum fills, by using excerpts of the captured footage fast-forwarding, re- winding and pausing to the drumbeat. As the street scenes were captured from the windows of public transport, the motion blur could not be separated from the objects in the image by masking or colour- keying. Given these restrictions, the synch-points were aligned by the motion that was resultant in the capture of media, which emerges from changes in the tracking of the footage. Synch-points were formed by synchronising with the playback functions: as the footage changes from one kind of tracking function to another, for instance, as the rewinding footage begins to fast-forward, the moment at which the shift occurs is used to accent a drum sound.
The motion of flowers and background video layers were aligned to the same rhythmic accents, which comprise the foreground video layers as short-envelope synch-points. To synchronise with rhythms in the music, the transparent layers of flowers gesticulate and fall through frame: their point of extremity in their motion being used as the frame to align with the attack of a drum sound. In establishing precise moments in the timeline at which to align to a drum strike, the footage of flowers falling in a linear direction was problematic due to having no mark or pinnacle in the gesture which could fuse with an accent, such as an impact or the trace of a morphology of movement. As such the linear motion through the frame without a peak made for less convincing short-envelope fusions.
75
As a final method of converging layers of footage, a level generator was used and the in-game motion from a first-person perspective were recorded, in which game models of furniture and skulls were positioned on the floor of a space. By treating this footage with a colour-key effect, the various animations, such as explosion and combustions were isolated and aligned to create collision synch- points. The high-definition screen-recordings of exploding objects in Garry’s Mod and scribbled lines in various painting apps fuse with short-envelope percussion sounds and staccato melodies in the song. At this stage in the process, colour-keying was also used to isolate screen-recordings of motion, by using screen-captures of explosion VFX against a green-screen in a level generator and Adobe Photoshop for recordings of brush strokes and scribbled line animation.
Figure 25: Mobile phone footage re-recorded onto videotape in School.
Figure 24: The layering of video footage in School.
76
Synchretic Transmutation
Instances of transmutation are those depicting “the action of changing or the state of being changed into another form”, in which a shift from within occurs in synchronism with a sound (Lindberg & Stevenson, 2011). This definition of transmutation is synonymous with morphing in the context of animation, as a production technique which refers to when “an image or model of one object is metamorphosed into another” (Hodgins, Obrien, & Bodenheimer, 1999, p. 687). In this text, I have used the term transmutation to refer to a morphing video effect in its various autographic and analogue/digital permutations, which includes any visual change in the form of an object or subject in the mise-en-scene from within. Although transmutation and directionality similarly exemplify synchresis to a long-envelope, dynamic synch-points are indicative of an external movement of an object around and in the mise-en-scene, rather than a change from within a form.
Figure 26: Animation depicting a hair product's benefits.
In commercial advertising, synchretic transmutation is used for consumer goods to “aid in self- realisation and self-improvement”, which enacts the “rhetoric of transformation” at the level of production (Kilyeni, 2012, p. 21). As a person or object mutates in the depiction of cosmetics, toiletries,
77
pharmaceutical and detergents, synchronisation is used in communicating a therapeutic rhetoric, as for instance medicine takes effect within the body, or as a cleaning product transforms a surface.
In music video, there are examples of transmutation which are conveyed from aligning the parameters of the sound effects, production of instruments and motion simultaneously. In the music video for Butterfly Effect (TravisScott, 2017, July 13), the shot displays a POV being submerged underwater, which occurs in time with the sound-effect of passing underwater as the song is submerged under a filter momentarily. As the shot is submerged underwater, several aspects of the visuals connote liquid properties, which are reinforced in the following shot by a distortion effect creating a wave pattern over a close-up of two women with the reflections of pool light on their faces. The example displays a triangulation of the music and sound effects and a willingness to disregard the music entirely for the purposes of a deviation of multiple elements of the audio tracks and the visuals which are taking on liquid properties.
Figure 27: The camera being submerged underwater in Butterfly Effect.
In the Fleet Foxes video for Mykonos (SubPop, 2009, Feb 11), the animated music video features construction paper cut-outs, that are combinations of shapes which resemble objects and animals. In a moment of transition as a cymbal hit rings out, the shape of a foot knocks against an indistinct pile on the ground, which levitates and spins into the form of a bird. The mutation lasts for the duration of the cymbal sound, which using stop-motion animation depicts a mutation in the form from within. While there is no strong relationship with the sound itself, the duration of the cymbal is used as the length of time for the mutation to occur in.
Figure 28: Animation to vocal harmonies in Big Time.
The music videos for Big Time (PeterGabriel, 2012, May 4) and Sledgehammer (PeterGabriel, 2012, Sep 42) both depict transmutation, which emerges via a range of animation practice as a basis for the 78
reflecting each song. In Sledgehammer, live-action footage of Gabriel singing is augmented by claymation, pixilation and stop-motion, which is most consistently reflective of vocal melodies through lip-synching. In an interview regarding the production of the video, director Stephen R. Johnson states that most shots in Sledgehammer were based on a technique he developed for lip-synch stop-motion. The animation-based construction of the video frame by frame which allows for the detailed mapping of elements to the music, required Gabriel to remain for 16 hours as the animation occurred around him (MrStephenRJohnson, 2010, Nov 17).
The lip-synching provides connection to the imagery residing in the lyrical content, though there are more disparate responses between the animations and the rhythm of the music. During Big Time (PeterGabriel, 2012, May 4), in a bridge of the song layered vocals descend on each other, beginning as the previous phrases are still being resolved. As the layered vocals are progressively stacked, their entrance and exit points are visualized by a sequence of shifting mud and soil. As the camera tracks along the ground on the shifting earth, the layered vocal harmonies are sung by mountains which form to reflect singing the vocal harmony and lose their shape again as the melody is taken up by another land-formation. The complex relationship to sound is reflected in the careful illustration of the vocal stacking, that is depicted by the incremental shifting of the claymation.
Painted images that are animated frame by frame present additional transmutation, which is reflected by the lyrical refrain “getting bigger”. As the lyrics “my car is getting bigger” are sung, the painted frames morph between different iterations of smaller to larger vehicles. This similarly occurs for the refrain “my house is getting bigger” in which houses increase iteratively in size, and as “my bank account” is added images of coins morph into stacks of notes. The morphing technique that is used across both videos has a synchretic basis for the timing of the shifts, which allows for ambitious shifts to long-envelope sounds.
Pap: Surfin’ Sluffs
The music video for Surfin’ Sluffs in the collaborative Pap project explores transmutation between a cassette recorded song to a video produced using a software-generated frame-interpolation effect. The video is an example of transmutation to a song that has been recorded from cassette as a stereo mix of combined layers, the separate layers of audio were not mutable beyond the initial recording and as such provide a prescient audio-track to form the basis for transmutation synch-points. The synch- points emerge from a morphing effect applied to a distorted videotape recording of the sun setting on the ocean, which is animated to shift with the song’s rhythm. To build a scene which could be distorted 79
using videotape, the computer-generated scene of the sun setting over the ocean was modelled and rendered in After Effects using a water and light plugin called Psunami. The computer-generated scene allowed for the manipulation of the compositional elements separately, by varying the camera angle, the lighting and the motion of the waves to the music.
The technical effect used is frame-interpolation, which is applied for the purpose of elongating the transition between still-images, or the frames of a video. The transmutation is formed via the use of the video image, which morphs and shifts direction in synchronization with the rhythm of the song. A synch-point occurs when a cycle of the frame-interpolation effect completes and the video ricochets from one frame to the next. As there is no directional motion inherently in the footage that can be used to align with a sound to create a synch-point, this ricocheting motion creates rhythm. Testing this process revealed the initial footage did not have substantial motion for morphing, which was added by creating a generation or copy. As each generation lowered the image quality, the texture and motion resultant was re-digitised and had a frame-interpolation effect applied to it in Adobe After Effects. Using this final iteration of the scene, the motion of the waves in combination with the distortion added by the VHS artefacts rhythmically shift in time with beat of the song.
The process of multitracking each instrument using a cassette machine meant each layer was improvised, and the five tracks were recorded separately. The instruments were multi-track recorded on a Yamaha MT8X tape recorder with a high-bias cassette tape, using a track each to record a Roland 404SX sampler for the drum beat and vocal sound-effect, a bass guitar and an electric guitar with an Ebow to create sustained notes. As the instrumentation was improvised during the recording, each instrument was added without planning the course of the composition, resulting in a simple, repetitious structure that is comprised of one section, though less mechanical than other performances as there are not duplicated or digitally quantised, and as such shift in and out of sync.
Division of Consonance and Dissonance
Barthes in discussion of detergents, draws attention to the various connotations of dirt and its removal, as he observes Omo advertisements have a therapeutic effect: “not only do detergents have no harmful effect on the skin, but they can even perhaps save miners from silicosis” (Barthes, 2012, p. 31). He additionally observes of detergent commercials that they typically connotate the message of a chemical removing the dirt: the product ‘kills’ the dirt” (Barthes, 2012, p. 31). Chandler asserts that for Barthes, advertising for detergents and soaps invoke the “conceptual frameworks in which cleanliness– dirt and godliness– evil were vertically aligned with science– nature. In other words, the 80
vertical alignments had been of dirt with evil and with nature”, of which “the connotations are of a chemical or mutilating type” (Chandler, 2007, p. 105).
Dualities are conveyed by the connotations of opposing types of synch-points, such as in a Surf detergent commercial from 1987 (Mc, 2014, Sep 22). As high notes accompany the product taking effect on a white shirt, via animations of the logo transforming and spinning into dotted lines. As an animation indicates the detergent taking effect, the images are accompanied by a cluster of sweeping wind noise and an upward scalar passage of high notes. The pairing used in the depiction of the effects of a product displays a positive shift in synchronism, to insinuate the transformative effects and
Figure 29: The depiction of a shirt being cleaned in a Surf commercial.
the calm brought on by the product’s completion of its effects.
In a commercial for All (northbaysports, 2019, May 20) dishwasher detergent, the sequence initially depicts a dishwasher of used dishes which undergoes a transformation. The commercial begins to a foggy and dark dishwasher with an ominous drone, as the narrator describes the annoyances of faulty products. A consonant shift occurs as the narrator states “now the good news”, as major arpeggiated chords enter the mix followed by a drumbeat. As the second reveal of the same shot is shown of the dishwater opening, high-end sparkling sounds accompany the reveal. Similarly to the Surf commercial, a consonant shift in the audio paired to the images implies the positive shift in the experience of the buyer.
In a 1990 hair loss commercial for Up John Dermatology Vision (mycommercials, 2007, Oct 6), the protagonist is a balding man inspecting himself in the mirror as he is spoken to by translucent doppelgangers. Twin versions of him appear to his left and right side, which dissolve into the scene to a synthesiser sound trailing upward in pitch. The doppelgangers are visually distinct: one is a man with
Figure 30: Video dissolves in a hair treatment commercial for UpJohn Dermatology.
81
hair, who appears as the narrator states “…between being the man he wants to be”, and his opposite, a balder version of the protagonist is introduced: “the man he could become”.
The scenario is reminiscent of the dichotomy between an angel and devil on the man’s shoulder, as no information is supplied about the advertised procedure, but instead focuses on a narrative around the protagonist’s choice to remain bald. To correspond with his dismissive hand gesture, the motion is aligned to a sinking electronic tone and the voice of the narrator stating: “…and you decide you want hair…” as his movement catalyses the disappearance of his replica a few paces behind in the doorway to the sound of a falling synthesiser note. The negative connotation of the synthesiser tone is synchronised to the dismissal of his balding counterpart, which insinuates the protagonist has made the choice to get the advertised treatment, and by doing so, solve the problem of his anxiety.
In a 1991 commercial for the Plymouth Voyager (takutaq, 2008, Mar 22), orchestral music plays as the car mutates from prior to successive prototypes, culminating at the most recent model of the car being advertised. The timeline depicts the manufacturing updates with a morphing effect to illustrate a chronology of the incremental developments. As the vehicle transforms, it is accompanied by a triumphant passage of orchestral music accented by the rushing sound of cymbals, to which each mutation is synchronised. The various prototypes are shown morphing from several angles: a shot of the bonnet, the rear windows and boot from a reverse angle, the steering wheel and dashboard, the chassis underneath the car, the engine, the backseats inside the vehicle, culminating in a wide-angle profile shot of the car as the narrator states “the evolution of the minivan is complete”. Transmutation is intrinsic to the commercial, as it distils the process of evolution to a sequential audio-visual technique.
Figure 31: Morphing used to depict successive vehicle prototypes.
A commercial for Strepsils (DN849, 2010, Oct 11; "Strepsils 'Bestmen' Television Advertisement,") demonstrates how the emotional resolve in a story arc is conveyed by transmutation. The short animation depicts a public speech, as the character addressing the crowd’s enflamed throat pulses red to a gurgling sound. Following this indication of a sore throat, the lozenges are miraculously revealed under a silver cloche to the sound of flute trills and metallic glissandos. As the camera’s perspective 82
shrinks to a microscopic view of the character’s throat, a shrill wind accentuates the materialisation of light and lens flares as the sickness is cleared away: rising to match the image of revitalised health with the sound of harmonic resonance. The transmutation of the visual mode in synchronization with a sound effect, is used to divide the state of wellbeing and sickness. Scott’s description of consonance and dissonance in an advertisement for Bayer Aspirin observes a similar trajectory, “As the announcer builds his case, the dissonant music almost imperceptibly becomes consonant. The people now are smiling, relieved. Bayer aspirin has done its work” (Scott, 1990, p. 228).
Figure 32: Still frames depicting transmutation in a Strepsils commercial.
Rose argues that commercials “use images of ‘science’ to suggest that their products can order, investigate or overcome nature” (Rose, 2016, p. 129). She continues, stating, “many adverts use images of ‘raw’ nature to confer apparently natural qualities onto their products, such as perfectibility, danger and obviousness”, noting that as the nature is ‘cooked’, or mutates from one state to another, it also changes its medium (Rose, 2016, p. 129). In a cartoon advertisement for Fruit Loops (BrookeFreeman, 2011, Jul 29), the mascot Toucan Sam approaches a “cherry berry tree” and picks a fruit from its branch. As the fruit is picked, it trembles and morphs into one of the grains of cereal. In the gesture of a transmutation synch-point, the cereal is hybridised with connotations of naturalness and of a simple unmediated translation from fruit to packet. The individual production of each kernel is
Figure 33: Animation depicting the transmutation of fruit into cereal.
83
insinuated as Toucan Sam is depicted mutating a single fruit to a grain one at a time, while the audible accompaniment implies a magical force in the process of transmutation.
The transformative moment is catalysed by a timpani and cymbal strike, followed by a rapid glissando of chimes and strings rushing upward, ultimately reaching a pinnacle on a sustained note as the cereal reaches its final state. The orchestration conveys consonance to depict the completion of the change: a plateau is reached in the music to sustain on a final note, as the cereal is held high above the cartoon bird’s head. The industrial production and distribution of the cereal and any additions of flavouring and sugar, are reduced to the cartoon mascot sourcing the cereal from nature. Presented in this way, the bird is a synecdoche for the cereal company: its representative and face, while the fruit’s transmutation is a metaphor for the production process.
Peter Bowman Jr.: An Arrangement
The music video for the Peter Bowman Jr song An Arrangement exemplifies synchresis between footage of coloured dye in motion to the gradual volume changes. The song which was recorded and sequenced in Ableton Live in the style of a 1980s new-wave synthesiser ballad, is comprised of melodies with slow-attack and release transients, or those which gradually crescendo and dissipate in volume, in contrast to the fast-attack transients of drums, pianos and guitar (Gallagher, 2009, p. 9). In response to the slow transients which characterise the instrumentation, the music video explores synch-points of dynamism and transmutation, which are visualized by footage of coloured dye drifting into prominence and receding suitably to match the music.
Figure 34: High-definition video footage of coloured ink in An Arrangement.
The instrumentation of An Arrangement is comprised of notes which gradually reach and recede from their peak volume, to coloured dye appearing and disappearing to and from the frame (Skinner, 2006, p. 78). To produce transmutation from the mise-en-scene at the time of recording, the dye was used to visualize the slow-attack and release of the melodies with a languid and gradual change, which over 84
repeated takes would result in variations of speed and form. The synch-points were then constructed in the editing stage by combining footage with instrumental cues, to find propulsion of the dye that would correlate suitably in duration with the length of a note in the song. The video layers of dye in motion are hierarchised with the volume of the foremost instrumental track in the song, which enter and leave the frame either by the volition of their own movement at the time of recording or via the animation of various effect values.
The isolated layers of dye were captured by using milk to supply a white background, with a luma-key effect applied to render all any white in the image transparent. Detergent and dye were then added to propel the dye across the surface of the milk and mix the colours. Given the process of capturing recordings of dye, each repetition of the colours being thrust along the surface of the milk would require careful resetting of the scene. As the dye languidly floats, its movement to the centre of the frame matches the volume envelope of a note, which responds analogously as the motion recedes to the gradual lessening of intensity in a musical cue.
The suitability of these fusions was achieved via trial and error, from the feedback process of watching the editing sequence and the footage and judging the motion required to match a musical cue. While the process of capturing footage was chaotic and reliant on chance, it was impossible to manufacture the dye’s motion which would occur by the chemical reaction captured at the time of recording. Given this, each recorded instance of the dye resulted in a unique blend of colours and gestures of motion. To capture extreme close-ups of the dye morphing, digital video footage was recorded with a Black Magic Cinema camera fitted with macro lenses for a Canon EF mount. To further reflect the attack and release transient of the instrumentation, each layer of footage was animated using keyframes, such as: the speed of the footage playback, position, scale, blur, and the movement of layer masks. The process was to align the keyframes of different video effects simultaneously, as the foreground layers wipe away the underlying layers in their movement from one side of the frame to the other. To remove underlying video footage, layer masks were added in Adobe Premiere Pro which were traced frame- by-frame using the pen tool, to assure a background layer’s disappearance in time with the foreground layer as a result of being eclipsed by its form.
The foregrounding of a video layer responds to the foremost instrument or note, by directing the audience’s eye to the layer most responsive with the melody, while obscuring melodies which recede from prominence. Synch-points are reinforced by the keyframing of various blur effects, in which the sharpness of a layer is lessened as a musical cue fading to silence. As the background video layers disappear behind foreground layers, their lessening in correspondence with the foremost instrumental idea are blurred to draw the viewer’s attention to the layer most imitative of the foremost musical
85
instrument or sound. The various efforts to animate a video layer into prominence or eliminate it from view are a standard practice of animation, during which “visuals can be brought into focus with carefully placed audio that directs the eye” (Beauchamp, 2013).
This is also exemplified in the short excerpts included in the omitted compilation of the Pap music video for I’m Gonna Block You, which presents the motion of microscopic cell photography fused with long-envelope notes. For the purposes of collecting directional motion of live-action footage that could be used in the creation of a synch-point, the micro-photographic close-ups of blood cells develop slowly to long-envelope instrumentation: slowly emerging take time to form and dissipate into the darkness. The music for I’m Gonna Block You was written using instruments with slow attack- transients, to explore the unison of long-envelope synch-points, in which a song’s instrumentation has sounds that reach their peak intensity gradually.
Figure 35: A magnification effect applied to a video recording of cells in I’m Gonna Block You.
With the aim of using the directional motion of the cells inherent in the shots rather than by key- framing, I found a synch-point could not be forged without isolating the motion of cells. This became an insurmountable problem with attempting to align a synch-point, as a motionless shot would detract from the impetus of a synch-point. The footage of the cell movement could not be isolated easily from other areas of the image, as the cells would merge into each other and move rapidly through the frame.
In shots where it was possible, Adobe Premiere Pro was used to augment the footage of the cells by isolating motion via magnification or masking, by changing the parameters of effects software in time with the music. For instance, while a synch point is fused to a flow of blood cells entering from one side of the image, another synch-point may be provided by a magnification of a portion of the image. At other points in the video, this is achieved by locating instances of a cell developing in isolation, then masking around its perimeter and playing it back in time with a synthesiser cue. While the excerpt 86
remains incomplete, the cells mutating to the arpeggiated melodies illustrate transmutation in a sequence.
87
Synchretic Dynamism
The suffix dynamic refers to synchronism between audio and visual tracks which are bound in motion, that depict a “trace of a movement or a trajectory” (Chion, 1994, p. 10). While both dynamic and transmutation synch-points are defined by a prolonged fusion or long-envelope, they are differentiated from instances of transmutation, as they trace the morphology of an objects motion rather than a morphing occurring from within. The term dynamic is borrowed from Chion’s (1994) discussion of fixed or dynamic sounds, which imply “displacement or agitation”, as opposed to those which remain “fixed” (Chion, 1994, p. 10). Fixed sounds imply stasis, such as the dial of a telephone or speaker hum, which Chion (1994) characterises as unique to their artificial origins (Chion, 1994, p. 10). Natural sounds may be fixed, such as when flowing water produces a likeness to white noise, although these sounds typically bear irregularity and the “trace” of motion (Chion, 1994, p. 10). As a contrast to short- envelope instances of synchronism, dynamic synch-points trace a maintained fusion between the images and sounds.
A dynamic synch-point refers to a sustained period of synchronism, that is prolonged between a musical cue or sound effect and the trace of a gesture or motion in the mise-en-scene. Dynamic synch-points are to be considered as discrete, single instances of maintained or continuous bond in the timeline. These sustained instances of synchronisation are fused with camera motion, light changes, and the direction of objects or people through the frame. As the audio and video develop in tandem during their sustained bond, dynamic synch-points give the impression of a single audio-visual recording, rather than discrete audio and visual elements moving in parallel. In returning to Birtwistle’s characterisation of morphology, the category of dynamism highlights examples which illustrate a “trace” or “trajectory”, in which “the parallel development of sound and image within a temporal frame” (Birtwistle, 2010, p. 190).
The discussion begins with an examination of Mickey-Mousing in the early animations of the Disney studio, followed by contemporary examples of the same technique in the work of Cunningham (2003). In commercial advertising, this section explores the sound effects accompanying logo animation, the substitution of a person’s voice and the replacement of an audio while maintaining the visual emitter of the sound. The artefacts are illustrative of scalar movements of melody, such as in Thousand Yard Stare between the motion of objects in the mise-en-scene synchronised to the direction of pitch, and as scalar movements are illustrated in The Heel, between the duration of a note and the movement of vehicles in the mise-en-scene.
88
Dynamic Examples of Mickey-Mousing
Synchronising the morphology of an image to musical expression is exemplified in the Disney studio’s Silly Symphonies (Disney, 2006), which were started in 1929 as vehicles for the studio’s exploration of “subordinated action to music” (Canemaker, 2001, p. 14). Jacobs characterises the Silly Symphonies as experiments in synchronization which showcase the new possibilities afforded by technology to precisely and inventively match sound and image (Jacobs, 2014, p. 65). As Brophy characterises them, these early experiments are similar in their depictions of musically fused gestures, in which synch-points emerge from the characters interactions: “expelled by the explosive percussiveness of their mauling of nature (animals, plants, the elements, etc.)” (Brophy, 1994, Para 23). The sounds emerge with musical performer and sound generator, which Brophy suggests are “a distillation of Western music’s traditional recourse to master nature to produce music” (Brophy, 1994, Para 23).
In the Silly Symphonies animation titled Musicland (Disney, 2006), the characters are anthropomorphised musical instruments that in their motion and speech are synchronised with specific instruments. As each anthropomorphised instrument greets their love-interest, a note phonemically imitates a “Yoohoo!” sound corresponding with the instrument type and gender; a high-pitched string phrase for the female violin, responded to by the male saxophone with a lower-pitched brass phrase. These anthropomorphised musical gestures represent the antagonistic character of villains that are conveyed through lumbering, lower-pitched musical phrases to imitate the bellowing laughter of villainous characters, or high pitched squeaky quivering musical cues for the nervous protagonist affirming his wedding vows. The narrative is emblematic of the early Disney studio: of a romanticised fairy tale between two lovers in the face of “grim and treacherous adversity”, in which songs communicate deliver positive and negative connotations through music, reflecting the “uplifting” and “disturbing” elements of the narrative (Coyle, 2010, p. 25).
Figure 36: Still frames from Musicland depicting the unfurling of a clarinet.
Mickey-Mousing is depicted in the opening scene as the protagonist unfurls a clarinet and uses it as a spyglass, drawing it to his eye to peer into the distance. This unfurling gesture is matched with a 89
suitably ascending clarinet phrase, which descends the scale as the spyglass recedes to its compact state. The synch-point illustrates a conflation in the audio-visual relationship of the iconic sign and the sound produced as a result of its function, between the image of a spyglass producing a clarinet phrase. This occurs as the protagonist leaps from the balcony onto a floating raft of xylophone bars, matched to a glissando sound. On the xylophone-raft, the protagonist rows across the river as the raft- planks collide back and forth in-synch to an ascending and descending scalar musical passage. The gesture is imitative of the scalar music, which fuses the animated motion of the characters and objects with an envelope aligned according to the direction of the pitch. In the example, the visual similarity of a spyglass and a clarinet, and the raft to a xylophone allows for the directional gesture to have a syncretic correspondence, displaying a complex conflation of the iconic and symbolic aspects of both objects.
The wooden texture of the logs imitate the wooden timbre of the xylophone, there is a scalar syncretic correspondence with each row of the gesture, in addition to the musical imitation of speech and suggestion of gender. While this brief example does not comprehensively reflect on Disney’s experiments, it is illustrative of the complexity of synchronisation achieved by the studio. Eventually the development of Disney’s style was freed from the tyranny of rhythm to an increasingly realistic style, which Jacobs attributes as beginning “with the departure of director and animator Ub Iwerks in 1930” (Jacobs, 2014, p. 72). While these cartoons from 1930 onwards “continued to ’match’ music and movement in some way”, “they experimented with more complex and realistic movement but, rather, developed more subtle and varied methods for synchronising the music and image tracks” (Jacobs, 2014, p. 72).
Mickey Mousing in the Work of Chris Cunningham
Although the breadth of Chris Cunningham’s (2003) output is not solely a focus on synchronization, a thread throughout his productions are the exploration of Mickey Mousing, that incorporates synchresis as an element in a breadth of multimedia practice, ranging from “short films, advertisements, art gallery commissions, installations, music production and a touring multi-screen live performance” (Leggott, 2016, p. 243). In a discussion of his process, Cunningham (2003) has stated that due to the images being reliant on the music for their tone, the music requires a similar sensibility, synchronism is determined in advance, by planning with detailed storyboards when and how motion will align with the music. The production methods are “closer to animation than editing… sometimes spending a day on just getting two frames to work to the music” (Records, 2005). This meticulous approach to synchresis is exemplified in the experimental short-film Rubber Johnny (Cunningham, 2005), during which 90
Cunningham’s focus was on pushing rapid synchronisation while maintaining coherence: “to see how far you can go before it becomes nonsensical, a mess… It was incredibly difficult to edit this video and find that line, where it seems breakneck, but still flows and makes sense as a sequence” (Røssaak, 2012, p. 176).
The experimental video Rubber Johnny (Cunningham, 2005) begins with a grainy close-up of an out of focus “grotesquely deformed man/child” played by the director himself in prosthetic make-up, who is answering questions to an off-screen interviewer (Leggott, 2016, p. 244). The moving images are captured in the grainy, green-tinted night-vision mode of a digital video (DV) camera, implying through the lack of careful framing that the content has not been manipulated or effected and is rather a document or found videotape of the interview process than fiction (Dawtrey, 2003, p. 20). The low- resolution digital video (DV) footage in combination with the application of various effects from Adobe After Effects and Photoshop allows for the addition of various gestalts in time with the music. The use of the liquify effect enables Cunningham (2005) to contort the protagonist’s anatomical form, while the beam effect allows for the placement of laser-beams which are emitted in and around the room in time with the music. The effects are keyframed, or carefully animated to appear and disappear in synch as music reacts with the environment and the character feverishly morphs his body in synchronised motion to the music.
Key moments of Cunningham’s music video for Squarepusher’s Come on my Selector (Cunningham, 2005) exemplify collision synch-points between human gestures to scalar musical phrases. In a 91
narrative that ranges in tone between suspense, horror, comedy and absurdism: two security guards attempt to prevent the escape of a young girl and her dog from the Osaka home for mentally disturbed children. Collision occurs between the movement of the guards in the mise-en-scene with the trajectory of the melody. As one of the guards attempts to flee down a stairwell, the impact of his footsteps descending each stair fuses with synthesiser notes of a downward scalar passage. As the guard reaches the bottom of the staircase, he pauses in surprise in synchronisation with a break in the music. As he changes his direction to reascend the staircase, the music re-enters and the synthesiser notes move up the scale to each footstep. The sequence depicts collision synch-points at the attack transient of each note to the guard’s feet hitting the stairs, which articulate a scalar correspondence between the direction of the musical phrase ascending or descending pitch to the direction of the guard’s movements on the staircase.
Figure 37: Movements up or down a staircase to a suitably ascending or descending musical phrase.
In a commercial for San Giorgio Pasta (Classic90sfan, 2018, Oct 22), dynamic synch-points are forged between motion in the mise-en-scene and musical accents. In communicating its durability, the pasta is bounced, thrown and squeezed in synchronisation to musical cues. As a slow-motion string of pasta is sent into orbit, its impact against a wall is synchronised to the sound of a cymbal crash. In another moment, a close-up of a fusillo being squeezed between a thumb and forefinger is synchronised to a scalar clarinet passage: ascending and descending on the scale in time with each squeeze of the pasta. The moving images are illustrative of the morphology of the sound in synchronism, which are expressive of the direction of pitch of the orchestral instruments.
92
Figure 39: Pasta being squeezed in synchronization with a scalar musical cue.
In addition to directional motion of the pasta fusing to a scalar idea in the music, the example illustrates a synch-point which enacts an audio-visual seal on the product. As the narrator states, “the medal for perfect pasta goes to… San Giorgio”, a hand enters the frame and places a medal onto the pasta packet, which transforms into the medal on the packet and glows. During this sequence there are two high-frequency bell-like chiming noises, which are musical accents occurring in synchronisation with the melodic resolution, the second chime entering on the final note the orchestra rests on.
Figure 38: A seal being placed onto packaging in synchronism with a sound effect in a San Giorgio pasta commercial.
Particular examples in advertising illustrate synchresis as the pitch of a melody ascends or descends in pitch and is completed with an accent, to depict audio-visual morphology in synchronisation to a point. In a commercial for Surf detergent (80sCommercialVault, 2016, Apr 13), as a spray bottle is filled up with solutions from two beakers the morphology of the sound trails upward, from a low to high pitch. This motion is accented by a sparkling sound at the completion of the bottle being filled, which like the pasta bottle is a kind of seal. As with the San Giorgio example, the gesture implies a commendation which the brand is placing on itself, that is communicated with brevity using a synch- point.
Figure 40: Light being used to accent the completion of a gesture in synchronisation in a Surf detergent commercial.
93
In a commercial for Burger King (StephenPayne, 2018, Aug 24) the performance of magic trick is accentuated by sound effects, as a tablecloth rises and is pulled away to reveal a stack of milkshakes underneath. The tablecloth rising to hold the shape of milkshakes is paired with an ascending sound, which is accented as the tablecloth is pulled away by the sound of chimes. A commercial for Kellogg’s low fat granola (80sCommercialVault, 2018, Aug 4), similarly demonstrates the pairing of a dynamic with an accent, as the sequence begins with the narrator discussing “the other side” of granola being fat content, the cereal packet turns around to reveal its label “granola” to “fat” on the other side. The packet suddenly falls through a hole in the table and out of sight. As the cereal box turns around, a whistle sound rises upwards, then as the cereal box is pulled through the table a timpani accent sounds to complete the accent.
The passages of dynamic synchronisation are reminiscent of research which is concerned with understanding music using verbally described meaning (Gabriel, 1978, p. 13). Particularly, in relation to Cooke’s (Cooke, 1959) claim that while ascending melodies of rising pitches energize the listener “to rally people behind a cause (for instance, national anthems)”, contrastingly melodies of descending pitch in allowing the vocalist to decrease their effort and be “more passive, inward-looking” (van Leeuwen, 2012, p. 321). Cooke states that ascending melodies, or those that rise in pitch are active and dynamic, relate to a greater vocal effort, as van Leeuwen asserts the higher the pitch of the note, the more effort is required from the vocalist (van Leeuwen, 2012, p. 321). Van Leeuwen asserts the same principle applies for intervals between notes, as larger more energetic upward steps typify heroism, while smaller steps downwards are “for ‘sentimental’ music and ‘ballads” (van Leeuwen, 2012, p. 321 and 322).
Thousand Yard Stare
In the omitted compilation of the artefact submission, the visualisation of each separate instrument simultaneously is illustrated in the music video for the Gary War song Thousand Yard Stare. To depict each instrument in synchronization to a responding visual element, the music video depicts a computer graphical user-interface (GUI), in which each instrument, beat and note comprising the instrumentation in the song would align to a different animation spaced evenly in a flat 2-D background. Although the excerpt submitted is incomplete due to the complexity of the idea and the lack of time to complete the video, it is the only submission in which the song has not been written by me and recorded under the guise of a moniker. Prior to my own recording experiments in the artefact submission, the song Thousand Yard Stare by Gary War was chosen for the complexity of its composition and for it being demonstrative of rapid scalar passages. 94
The excerpt included in the omitted compilation depicts a GUI, as the animation proceeds by way of images on a computer screen of data acquisition, medical imagery and diagrams, cross-section of anatomy, symbols and graphs. The visual style of a GUI is intended to give an impression of analysis and measurement from the rapid gesticulation of graphics which imply evaluation is occurring. While the visual style is reminiscent of the convention of interfaces used in film and television to elicit an emotional response, the GUI has no function other than to visualise the instrumentation. The process of animating began in pre-production by diving each section of the song into the number of images necessary to collect for each note. In the preparation for animating a sequence of still images, the instrumental track to which images were synchronised would be isolated, allowing for the playback of each instrument. This would allow for the precise placement of synch-points, to articulate rapid passages of melody that were obscured between the other instrumental tracks in the song’s mix.
Figure 41: Instrumentation assigned to different animated images in Thousand Yard Stare.
With the aim of depicting all the instrumentation simultaneously rather than synchronise to a dominant instrument, melodies bubble, distort and overlap as arpeggiated scalar passages of melody cycle between extremities of pitch. Consequently, a melody becoming louder or raising in pitch is synchronised to an image increasing or diminishing in scale or a change in the hierarchy of the image- sequence. This is illustrated as an image sequence of guns are animated to a downward arpeggiated melody, to which each note further down the scale the responding image of guns consecutively diminish in size, changing from rocket launchers, to machine-guns, to pistols. As the scene shifts to a medical theme, a sequence of anatomical illustrations depicting the age of a person being older or younger responds to the pitch moving up or down the scale respectively. These are examples of
95
dynamic synch-points that respond between the scale of the object and the hierarchy of the content, to upward or downward moves in pitch of the following note.
During the planning stages, a diagram was made to visualise the requirements of the layers of instrumentation and the amount of still frames necessary to comprise each melody and rhythm. As illustrating each rhythm and melodic change in the music required a large amount of raw material or footage, it was necessary to count the number of beats for each instrument. In the diagram, each image is representative of an aspect of the instrumentation, each instrument the number of images needed has been accounted for, which splits the song into separate categories, denoted by the structure of the song. In the simple sections of the song, four images may only be needed to represent a ten second period, while in other areas there are twelve individual images occurring per second.
Figure 42: A table depicting the planning to assign instrumentation with animation.
Following the collection of raw material, each image sequence was given the same pixelated aesthetic using a series of actions in Adobe Photoshop. This process would reduce any image to a limited number of pixels and colours and give a consistency to the range of content and aesthetic styles of the sourced imagery. While the video remains incomplete, the excerpt displays synchronization to scalar passages of melody, and of each note and beat in the song being simultaneously depicted.
Incongruous Dubbing
Incongruous Dubbing is indicative of synch-points between a visual source that is different from that of the sound produced: as a person speaks with a different voice from their own, such as when an animal or child speaks with an adult voice. The fusion relies on carefully tracing the timing of an existing
96
recording, as the synchronisation can begin with either a prescient video track as the guide for replacement audio or by miming to an audio recording. As the imitation of the envelope creates fusion between the spoken word and the image, the imitation of the prescient mode produces a seemingly inextricable link. In its likeness to the rhetorical figure of substitution in advertising, incongruous dubbing can serve a comedic function by depicting animals who talk and reversing the speech of adults and children, it is also discussed herein as a gesture to replace the musicians as the source of a song (Durand, 1983).
In the music video for The Arrows’ Talk Talk (mrbriefcaseMUSIC, 2016, Jun 7), incongruous dubbing undermines the musicians as the locus of the musical performance, by distributing the lyrics of the song among characters in the scene. The lyrics are sung on a futuristic set encircled by people dressed in silver uniforms, robotically marching around white curtains and glass block windows to the beat of the song. The various choreographed gestures are timed to the rhythms: as for instance the receptionists chant the lyrical refrain and close-ups of typing is synchronised. This kind of choreography is commonplace in music videos, in which objects and people in the mise-en-scene are arranged in pre-planned choreographed motion to the music.
Figure 43: Dancers singing the refrain in the Talk Talk video by The Arrows.
In the Talk Talk music video, there is synchronisation in the scene from the choreography and the lip- synching. In combination with cutting, other examples illustrate dynamic synch-points across a sequence of shots. For instance, in the music video for Fireflies (OwlCity, 2009, Dec 15) a sequence of lights turn on in synchronisation with the melody, which is catalysed by the performer pressing a “magic” button on the keyboard. The sequence is synchronous from various sources: as some synch- points are established by the lights turning on, other cutaways show the performer striking the keyboard in time with the notes. The sequence ends with what sounds like white noise rising in pitch and ramping up in speed. The envelope of this upward moving trajectory is matched with the image of a vinyl record spinning, which has been manipulated to ramp-up in speed to fit the trajectory of the sound.
The Kids Voices commercial for Haribo by Quiet Storm (UKHaribo, 2018, 26 July), is premised on the incongruous dubbing of children and adult voices. As the commercial begins, a man leans from the
97
back of a crowded elevator to offer around some confectionary, stating in a child’s voice “New Haribo Fruitilicious are delicious!”. Each of the adults accepts, similarly speaking in children’s voices as they discuss the flavour. In their enthusiasm, they are distracted as a child in the crowd interjects, clearing his throat and speaking in an adult’s voice, “If I could just draw your attention to the packaging, you’ll observe these have thirty percent less sugar”.
This idea is also used in the Haribo commercial for Starmix, in which people in a board meeting speak with children’s voices while sharing the confectionary. The incongruency suggests the reversal of minds and social roles, as adults are given the mentality of a child and vice versa. The reversal of roles is insinuated by the adults speaking in children’s voices and vice versa, which is reflected in the trajectory of the conversation as the adults make naïve observations and wishes about the shape of the candy, while the child is the only conscientious person in the scene.
Figure 44: A Haribo commercial depicting incongruous dubbing.
Shreds and Musicless Music videos exemplify incongruent dubbing in user-generated content native to video streaming platforms. These videos subvert the original music video by redesigning or replacing the audio of the official promotional material with a “user-generated meme and parody version” of an existing video (Sánchez-Olmos & Viñuela, 2017, p. 3637). In both Shreds and Musicless Music videos, the performative gestures of the original music video are the prescient mode, which guide the alignment of new audio. The replacement of the original audio undermines the production value, as the gestures are undermined by unspectacular, dissonant re-recordings of a song, or by a lack of accompanying music. In the precise tight-sync to a palette of new sounds and their careful alignment to the motion inherent in the video, the impression of awkwardness and ineptitude becomes inextricable from the performers.
Musicless Music Videos are user-generated parodies of existing music videos, in which the music is replaced by foley sound, or sound effects in synchronisation to the actions of the on-screen performers. For instance as a performer dances, the music which usually accompanies them is replaced by the sound of the material of their clothing and the clatter of their shoes. In Wienerroither’s (2016) work, who has created some of the most popular iterations of the Musicless Music video (MarioWienerroither, 2014, Jul 8), the cloth pass or material of a person’s clothing is audible, as well as their breath and the squeak of their footsteps. As a compositional practice, Dotto observes the
98
enacting of a “reverse Mickey-Mousing effect” in the muting of the original soundtrack: “unravelling the unheard sonic dimension of images we may remember as strictly associated with music” (Dotto, 2016, p. 26 and 29).
Musicless Music Videos and Shreds videos are similar in their foundational idea that force ineptitude onto a musical performance by synchronising an existing musical performance with unskilled and dissonant replacement musical tracks. In Shreds videos, the audio-track of a musical performance or music video is replaced with an ineptly performed rendition of the same song that originally accompanied the video prior to its subversion. Characteristically of UGC that is native to streaming platforms, the humour is accentuated by ineptitude. The effectiveness of this incongruous dubbing avoids disregard by adhering to the envelope of a visual gesture, to undermine the visuals with underwhelming sounds.
The audio which replaces the original soundtrack is aligned to performative gestures, such as guitar strumming, drum hits and lip-sync, although the sounds are dissonant and clash inharmoniously, subverting the aim of lip-sync: “the goal of lip sync is for it to be as unnoticed as possible” (Garwood, 2012, p. 42). The example pictured satirises a live performance of I Get Around by The Beach Boys (massbetelnut, 2012, Nov 12), in which the original song is replaced with out-of-tune harmonies and a flatly performed and produced re-recording of the song. The replacement of the song in synchronization subverts the power of the image with an inept performance, while maintaining an inseparable visual link between the performance and the audio track.
Figure 45: A performance by The Beach Boys used in a shreds video.
The Fed-Up Kids: The Heel
In the music video for The Heel, Mickey-Mousing is illustrated between short-envelope notes and the entry and exit of vehicles and pedestrians through the mise-en-scene along the central business district of Melbourne, Australia. The captured footage from a tripod depicts the motion of vehicles and pedestrians entering from one side of the frame and leaving the opposite side: as short accented notes are depicted by cars, buses or trams provide the visual alignment for the long-envelopes or sustained
99
sounds. Each vehicle passes rapidly through the frame for the duration of each note, to create a synch-point with the attack transient (the starting sound of a note) and duration (length of the note) of the melody. In illustrating the musical shift of the final section of the song, the video also explores scalar correspondences between pitch-height and the motion of an object through the frame between the melody with the position of an object in the frame.
While the adjoining The Fed-Up Kids music video for School divides synch-points across layers of visual recording media, The Heel makes use of footage from a National M1 videotape tripod-mounted camera and uses the vehicles and pedestrians as the only motion in the mise-en-scene. This method was to make the alignment of synch-points seem to emerge from the unedited playback of surveillance footage, rather than a bricolage of different video formats aligning with the playback of the song. Due to the fast melody and brief intervals between a note and the following note, the time restriction necessitated the motion to pass through the frame rapidly, as to not interfere with the previous note or have two vehicles in the same position at once in the mise-en-scene.
To create synch-points with the presciently created music track of a lead melody being reinforced by multiple instruments, the vehicles and pedestrians moving through frame were sped-up or slowed- down to reflect the length or duration of each note. These alignments are maintained throughout the song from start to finish, requiring each note to respond with a person or vehicle in synchronization. As the footage of vehicles and pedestrians moving through the frame has no peak or point of convergence, there is no certain frame for a synch-point to be aligned, as might occur with a collision, cut or strobe light used as a visual parameter.
The synch-points reflect moments of unique melodic variation, or of the occurrence of musical cue that is singular in the playback of the song. As the bridge section moves into the chorus, a synthesiser melody has a delay effect applied to it, which duplicates in synchronisation with the melodic phrase. To reflect the repetition of the melody, an echo effect has been applied in After Effects to repeat the motion of a pedestrian crossing the road. This similarly occurs in the sequence during a drum fill of sixteenth notes, that is fused to a cutaway of a malfunctioning streetlight strobing on and off rhythmically with the snare drum.
At the end of the video the setting shifts from day to night, as the cars are removed entirely leaving the isolated headlights in motion against darkness. The lights in motion are fused to comparatively languid and slower melodies than the previous section, to which a headlight’s movement through the frame is fused for the duration of each note. During this section the dynamic synch-points trace the scalar correspondence between the position of the lights and pitch-height of the notes: as a downward movement of a light is matched with a descending pitch, or an upward moving note with an ascending 100
pitch. In synchronization, the shift in the time of day reflects the structural and instrumental changes, during which the notes lengthen from staccato accents to sustained notes. As the scalar movement of a note is represented by the upward or downward movement of the car headlights, the envelope of each synch-point was achieved by slowing and speeding up the motion of vehicles through the darkness of the frame.
Figure 46: Headlights in darkness during the final section of The Heel.
101
Conclusion
The early stages of this text proposed deconstructing the form of a synch-point in a taxonomy, to identify and classify the constituent aspects of audio-visual synchronization in screen production. A taxonomy of audio-visual synchronization requires articulating how synchresis is comprised of nested elements: which identifies when a fusion occurs in the timeline, for how long it occurs and which formal elements of the audio-visual are changing in intensity. Using Chion’s discussion in Audio-Vision as a starting point, in which each single instance of audio-visual synchronism occurs in a synch-point, as a “sound event” and “visual event” align or “meet in synchrony”, I have suggested synch-points are comprised of varying durations and intensities, or modal parameters, the rising and falling intensities of which are contained together in a shape, or synchretic envelope (Chion, 1994, p. 58). While I have attempted to rigorously investigate a synch-point of different modes to each other, as is typical of practice-led research, the outcome can be seen as one of “problematizing traditional conclusions” in the attempts to classify instances of audio-visual fusion (Eisner, 2008).
In classifying the varying lengths of synch-points, a synchretic envelope is divided into two types, short long envelopes, to delineate between an audio-visual fusion as accented or sustained. In the discussion of examples of short and long envelopes in production, I proposed four synchretic operations: transmutation, directionality, collision and cutting, which are identified in the sites of music video and commercial advertising and form the basis for distinct video experiments in the production of the artefact submission. To be applicable to any instance of a synch-point, these broad classifications were formed to be inclusive of differences of the duration of a fusion: between a variety of acoustic and visual formal elements, such as object animation, the cinematography, editing of the sequence. In addition, I have considered the recording and playback media and how the addition of noise and distortion is an expressive aspect to particular synchronistic ideas.
To discuss instances of audio-visual synchronism that are transduced from one mode into another (i.e. from video to sound, or from sound to video), the process entails a translation from audio and video to writing. As is typical of practice-led research, the written component or linguistic explication, music and music videos were developed simultaneously, by returning to the writing through the duration of working on the artefact submission (Gibson, 2018, p. xii). The majority of writing occurred after an instance of synchresis was constructed, as the process of excavating the practice in writing which is typical of practice-led research, occurred during post-event analysis, or “analysis after the ‘event’ of creating”, which typically produces knowledge of the artefact, though is not elucidating of the practices themselves (Bubb, 2012, p. 47).
102
As the project is focused on audio-visual production rather than reception it was problematic to find methodology of the audio-visual, of which sound had “equal aesthetic weight” to the images and terminology of film analysis (Branigan, 1985, pp. 317-318). Finding a suitable methodology to extrapolate meaning from audio-visual synchronism was also fraught with inconsistencies, as writing is yet another means of translation, or transduction, in which linguistic metaphor and rhetoric are used to describe the synchronization of an audio-visual gestalt, which had been created one mode and translated to another.
In the various attempts to apply to the “equivalent of grammar” or the “inflected relation” and “syntax” or the “functional relation” as in speech or writing, Bertrand and Hughes contend “none has managed to account for all the elements in such a complex spatial and temporal field, which is not divided neatly into clearly bounded signs” (Bertrand & Hughes, 2017, p. 209). Synchresis is often executed rapidly and intuitively with little forethought, as Chion observes music videos in their production are not necessarily intellectualized or concerned about combining devices that might be “opposed in the abstract” (Chion, 1994, p. 167). Conversely, articulating the execution of a synch-point in linguistic terms must be wrought through a process of translation into writing, to intellectualize the emotional connotations and interplay of formal elements at the service of a message, distilled to a single or series of audio-visual gestalts.
The taxonomy I have suggested becomes problematic as the classifications expand, particularly at the level of synchretic parameters. This is due to a synchretic envelope not typically formed between one audio and visual parameter that are pitted in an equal ratio (i.e. pitch height to colour intensity), but as layered, unbounded relationships of multiple audio and visual parameters aligning to each other. The complexities which arise at the level of synchretic parameters can be illustrated by how the keyframing of a visual effect on a video track or audio effect on an audio track occurs in a video or audio editing software such as Adobe Premiere Pro or Ableton Live. Within Adobe Premiere Pro, for instance, the choice between a range of effect categories can augment audio or video information, for instance to colour correct, distort, stylize an audio or video recording. Accurately discussing each of the minute and incremental changes that abound in a create work is problematic, as it is rarely a single aspect of a plugin, but of multiple parameters that are not in equal ratio.
Considering a synch-point in isolation does not account for fusions that occur as a “chain” of other synch-points. A limitation of the taxonomy is this evaluation of synch-points gaining significance from the synch-points around them in a timeline, or a discussion of synch-points in sequence which may gain significance in relation to the synch-points which have occurred previously. Although synch-points are spotted disparately through the course of a film or video sequence, their significance is often
103
apparent by their relationship to the spotting of other synch-points in constellation. As Bertrand and Hughes note that “every sign is placed in a text syntagmatically, in relation to other signs (a relation of presence)” (Bertrand & Hughes, 2017, p. 205).
A discussion of the synchretic envelope of a synch-point is evaluative of the parameters which are fused, while ignoring the syntactical relations to other points of synchronisation. A synch-point considered in isolation may provide insight to the modal parameters in an instance of synchronisation, though does not explain the significance of its relation to the surrounding synch-points in the sequence, as one of many spotted moments of synchresis in a timeline. As well as the significance of repetition, this is especially important in the visualization of melodies, for instance, in the excerpt of the Thousand Yard Stare music video, as a scalar synthesizer melody arpeggiating downward is synchronized to an animation of incrementally lowering bars on a graph. This suggests classifications of the taxonomy might be more illustrative if applied at a different stage in the hierarchy, for instance perhaps the synchretic envelope should account for the entire passage of a gestalt, for instance, taking into consideration a cycle of a fire alarm rather than a single beep.
Parameters are not typically polarised between two extremes, or two aesthetic opposites, but instead as is typical of an effect in Adobe Premiere Pro, in which the interaction design presents a range of properties nested within categories, the values of which are often attenuated or boosted. For instance, to use the example of pairing of the parameter of brightness to the parameter of pitch, is rarely how they are produced in constellation with other synch-points and discounts synchronization between more than a single parameter for each mode. As parameters reinforce the same synch-point, the amount of audio tracks synchronising to video tracks are not necessarily in ratio to the prescient mode. Consequently, in taking the example of a single audio parameter, an instrument’s pitch rising, might fuse with brightness, glow, blur, saturation, zoom, or any other number of formal elements of the moving image.
In adhering to the use of the synchretic operations as the basis for creating synchronization, I found the music or music video production did not necessarily benefit by the distinctions established in the taxonomy. It was inconsistent and problematic to strictly assign instances to the basic definitions of dynamism, collision, transmutation and cutting, as often an idea outside the linguistic definition of the synchretic operation would emerge from the desire to synchronize with a prominent melody, though doing so would abjure the music video’s solely articulating a particular synchretic operation.
As for instance, in synchronizing to the most prominent instrumentation throughout a song, the shift from a slow-attack such as a string section, to a plucked guitar melody of fast-attack, the synchretic operation would shift from one of the long-envelope categories of transmutation or dynamic 104
operations, to a short-envelope variation such as cutting or collision. In addition, the examples of synchretic operations in commercial advertising and music video were rarely pitted in equal ratio, and the audio-visual synchronization could occur in a 10:1 ratio, as rarely being dedicated to one kind of formal gesture.
To experiment with different kinds synchretic operations, in this investigation the recording methods between analogue and digital media and discussed how each has affected the creation of a synch- point. The purpose of which was to explore the restrictions imposed to the process of synchronisation: of aligning a synch-point, as the differences in the medium-specific noise, or the “distinct sensory qualities that are direct products” of their “materiality and technology” (Newman, 2014, p. 32). The combination of noise and distortion depicted simultaneously in both audio and visual tracks is central to the artefact submission, particularly in Surfin’ Sluffs, L’Efforel, Cindy Doll, Exhausting Moment and The Heel. In the aim of separating each recording moniker with a characteristic musical production style, the analogue media were used for their medium-specific noise: analogue samplers, multi-track and reel-to-reel tape recorders, low and high-bias cassette tape, analogue synthesizers, quarter-inch reel-to-reel tape, VHS and VHS-C tape.
For synch-points that were not recorded simultaneously, there was an inherent need for a prescient or guiding track. The recording media would often dictate whether the audio or video track would be the prescient: that which had been created in advance of the other and would guide the expression of the synch-point. Recording using analogue media meant some audio or video could not be returned to or changed, as various formal aspects of an arrangement would become less modular in key stages throughout its production. For instance, in Surfin Sluffs or the It’s The Delmer Brothers! commercials, the re-photography of the digital images meant the separate formal elements of the scene could no longer be altered, which had previously been modular or susceptible to change in isolation, prior to given expressive noise via being transferred to VHS tape. In such instances, the “locking off” of a track is a defining stage of the work, as it is the catalyst for anything that will be produced in the following mode.
The materiality and distinctive means of engaging the recording media would determine the complexity of the recordings and the duration of the process, as is evidenced by examples of synch-points that have been achieved rapidly, and those in which have been designed and reinforced by a range of video or audio parameters simultaneously using careful design. The digital workspace allowed for careful arrangement via a visual interface as the use of Ableton Live or a DAW in the production of could be painstakingly revised, repositioned, balanced, attenuated and replayed endlessly, in a non- degradable workspace. Adversely, the utilization of cassette or quarter-inch tape introduced entropy
105
with repeated playback, layered recordings and duplication and the lack of an interface beyond a timecode did not facilitate the simple deletion of segments, and fixing of errors, often encouraging the recording of simple melodies and song structures.
As a pervasive element of amateur and professional screen-production, synchresis has a history stemming from the burgeoning experiments of audio-visual fusion that continues to develop in contemporary screen-production. The most obvious finding to emerge from research project is how limited this area is in research, that particularly practice-led studies which deal with how meaning is imparted at a micro level in a synch-point. The significance of this project is in the practice-led experimentation of synchresis with a range of musical genres and visual aesthetics, which presents the prescience of either audio or video as the mode. The incorporation of range of technologies and aesthetics explores the differences between the types of synchronization, that necessarily are viewed and the importance of which is seen and can only be wrought with at the level of production.
106
Reference List
8thManDVD.com™CartoonChannel. (2014, Mar 9). Classic G.I. JOE: Hasbro Toy Commercials (1980s) (Remastered) (HD 1080p) [Advertisement]. Retrieved from https://youtu.be/sRnXYV2eA7w. 80sCommercialVault. (2016, Apr 13). 80's Commercials Vol. 425 [Advertisement]. Retrieved from https://youtu.be/Uq20kqNqmOE?t=273. 80sCommercialVault. (2018, Aug 4). 90's Commercials Vol. 244 [Advertisement]. Retrieved from https://youtu.be/9NSVU4Gv_wA?t=513. 513official4. (2011, 19 Mar). Coast Soap 1989 [Advertisement]. Retrieved from https://youtu.be/aAS_SI-g6Sc. a-ha. (2010, Jan 7). a-ha - Take On Me (Official 4K Music Video) [Video file]. Retrieved from https://youtu.be/djV11Xbc914. Aikin, J. (2007). Envelope Generators. Keyboard, 33(7), 50-50. Allan, B. (1990). Musical Cinema, Music Video, Music Television. Film Quarterly, 43(3), 2-14. doi:10.2307/1212631 Allan, D. (2015). This note's for you : popular music + advertising = marketing excellence (First edition.. ed.): New York, New York 222 East 46th Street, New York, NY 10017 : Business Expert Press. Allen, R. (1995). Projecting Illusion: Film Spectatorship and the Impression of Reality. Cambridge: Cambridge University Press. Almeida, D. (2009). Where have all the children gone? A visual semiotic account of advertisements for fashion dolls. Visual Communication, 8(4), 481-501. doi:10.1177/1470357209343374 Alpert, J. I., & Alpert, M. I. (1990). Music influences on mood and purchase intentions. Psychology and Marketing, 7(2), 109-133. doi:10.1002/mar.4220070204 Altman, R. (1980). Moving Lips: Cinema as Ventriloquism. Yale French Studies(60), 67-79. doi:10.2307/2930005 Altman, R. (2001). The Living Nickelodeon. In R. Abel & b. ProQuest issuing (Eds.), Sounds of Early Cinema: Bloomington : Indiana University Press. Altman, R. (2005). Silent film sound. 462. Andersen, T. S., Tiippana, K., & Sams, M. (2004). Factors influencing audiovisual fission and fusion illusions. Cognitive Brain Research, 21(3), 301-308. doi:10.1016/j.cogbrainres.2004.06.004 Areni, C. S. (2003). Exploring managers' implicit theories of atmospheric music: comparing academic analysis to industry insight. Journal of Services Marketing, 17(2), 161-184. doi:10.1108/08876040310467925 Arnheim, R., Benthien, B., Newbury, C., & Diederichs, H. H. (1997). Film Essays and Criticism: University of Wisconsin Press. Aufderheide, P. (1986). Music Videos: The Look of the Sound. Journal of Communication, 36(1), 57- 78. Avanzini, F., Rath, M., & Rocchesso, D. (2002). Physically-based audio rendering of contact (Vol. 2, pp. 445-448). USA. Bak, M. A. (2015). "Ten Dollars' Worth of Fun": The Obscured History of the Toy Magic Lantern and Early Children's Media Spectatorship (Vol. 27, pp. 111-134): Indiana University Press. Balides, C. (2000). Virtual Spaces and Incorporative Logics: Contemporary Films As "Mass Ornaments". MIT Communications Forum. Retrieved from MIT Communications Forum website: http://web.mit.edu/comm-forum/papers/balides.html Barrier, M. (2003). Hollywood Cartoons American Animation in Its Golden Age. Oxford. Barthes, R. (1982). Camera Lucida: Reflections on Photography (R. Howard, Trans.). New York: Hill and Wang. Barthes, R. (1985). The Responsibility of Forms: Critical Essays on Music, Art, and Representation (R. Howard, Trans.). New York: Hill and Wang.
107
Barthes, R. (1988). The semiotic challenge: New York : Hill and Wang. Barthes, R. (2012). Mythologies : Complete Edition: Hill and Wang Farrar Straus & Giroux, C/O Mps, 16365 James Madison Hwy, Gordonsville, VA, 22942. Batty, C., & Kerrigan, S. (2018). Screen Production Research : Creative Practice As a Mode of Enquiry: Springer Verlag Springer Verlag, 1120 Rte 22 East, Bridgewater, NJ, 08807. Baudrillard, J. (1983). Simulations. New York: Semiotext(e). Baudrillard, J. (2010). The Masses: The Implosion of the Social in the Media. In S. Thornham, C. Bassett, & P. Marris (Eds.), Media Studies : A Reader. New York: New York Univ Pr. Bazelon, I. (1975). Knowing the Score: Notes on Film Music: Van Nostrand Reinhold Company. Beasley, R., & Danesi, M. (2002). Persuasive signs : the semiotics of advertising: Mouton de Gruyter De Gruyter, C/O Triliteral Llc, 100 Maple Ridge Dr, Cumberland, RI, 02864-1769. BeastieBoys. (2009, 15 Sep). Beastie Boys - Intergalactic [Video file]. Retrieved from https://youtu.be/qORYO0atB6g. Beauchamp, R. (2013). Designing sound for animation (2nd ed.. ed.): Burlington, Massachusetts Oxfordshire, England : Focal Press. Bell, D. (2006). Creative film and media practice as research: In pursuit of that obscure object of knowledge. Journal of Media Practice, 7(2), 85-100. doi:10.1386/jmpr.7.2.85_1 Beller, H. (2011). Between the Poles of Mickey Mousing and Counterpoint. In D. Daniels & S. Naumann (Eds.), See This Sound : Audiovisuology: Distributed Art Pub Inc Distributed Art Pub Inc, C/O Perseus Distribution, 193 Edwards Dr, Jackson, TN, 38301. Bender, G., & Druckrey, T. (Eds.). (1994). Culture on the Brink: Ideologies of Technology (Vol. 9). Seattle: Bay Press. Bergman, T. (2018). American Television: Manufacturing Consumerism. In J. Pedro-Carañana, D. Broudy, & J. Klaehn (Eds.), THE PROPAGANDA MODEL TODAY (Vol. 8, pp. 159-172): University of Westminster Press. Bernardi, J. (2001). Writing in Light : The Silent Scenario and the Japanese Pure Film Movement. Detroit: Wayne State University Press. Bertrand, I., & Hughes, P. (2017). Media Research Methods : Audiences, Institutions, Texts: Springer Verlag Springer Verlag, 1120 Rte 22 East, Bridgewater, NJ, 08807. Betancourt, M. (2007). A Taxonomy of Abstract Form Using Studies of Synesthesia and Hallucinations. Leonardo, 40(1), 59-65. Betancourt, M. (2017). Synchronization and Title Sequences : Audio-Visual Semiosis in Motion Graphics: Taylor & Francis Taylor & Francis, C/O Kentucky Dist Center, 7625 Empire Dr, Florence, KY, 41042. Beynon, G. W. (1921). Musical presentation of motion pictures: New York: G. Schirmer Boston: G. Schirmer. Bien, N., Ten Oever, S., Goebel, R., & Sack, A. T. (2012). The sound of size. NeuroImage, 59(1), 663- 672. doi:10.1016/j.neuroimage.2011.06.095 Birtwistle, A. (2010). Cinesonica : Sounding Film and Video. Manchester, UNITED KINGDOM: Manchester University Press. Björnberg, A. (1994). Structural relationships of music and images in music video Pop. Mus., 13(1), 51-74. doi:10.1017/S026114300000684X Black, J. (2016). The Thrill Ain't Gone: Eclectic Insights into the Evolution of Record Cover Art. In J. Austin (Ed.), Spinning Popular Culture As Public Pedagogy : Critical Reflections and Transformative Possibilities. Dordrecht: Brill | Sense. bluefacebleedem. (2020, Jul 9). Blueface - Yea Yea ft. Coyote [Video file]. Retrieved from https://youtu.be/YLH2k_oYJqk?t=38. 108
Bolter, J. D., & Gromala, D. (2005). Windows and Mirrors: Interaction Design, Digital Art, and the Myth of Transparency. Cambridge: The MIT Press. Bolter, J. D., Grusin, R. (1999). Remediation: Understanding New Media. Cambridge: The MIT Press. Bond, H. (2012). Lacan at the scene: MIT Press Mit Pr, 5 Cambridge Ctr, Cambridge, MA, 02142. Bonde Korsgaard, M. (2012). Creation and Erasure: Music Video as a Signaletic Form of Practice. Journal of Aesthetics and Culture, 4, e1-E8. Bordwell, D. (1985). Widescreen Aesthetics and Mise en Scene Criticism. The Velvet Light Trap: Review of Cinema, 21(Summer), 18-25. Bordwell, D. (2007). Do filmmakers deserve the last word? [Blog post]. Retrieved from http://www.davidbordwell.net/blog/?p=1174 Bordwell, D. (2012b). Pandora’s digital box: Art house, smart house [Blog post]. Retrieved from http://www.davidbordwell.net/blog/2012/01/30/pandoras-digital-box-art-house-smart-house/ Bordwell, D., & Thompson, K. (2012). Film Art: An Introduction (10th revised ed.). New York: McGraw- Hill. Bottomore, S. (1999). An International Survey of Sound Effects in Early Cinema. Film History, 11(4), 485-498. BoyMeetsGirlVEVO. (2014, Feb 21). Boy Meets Girl - Waiting for a Star to Fall (Official Video) [Video file]. Retrieved from https://youtu.be/RhxF9Qg5mOU?t=99. Brand, S. (1987). The Media Lab: Inventing the Future at MIT. New York: Penguin Books. Branigan, E. (1985). Color and Cinema: Problems in the Writing of History. In B. Nichols (Ed.), Movies and Methods (Vol. II, pp. 121-143). Berkeley: University of California Press. Branigan, E. (1989). Sound and Epistemology in Film. Journal of Aesthetics & Art Criticism, 47(4), 311. Breinbjerg, M. (2007). Sound in Digital Media: Symbol or Phenomenon? . The Centre for Internet Research, 10(Lev Manovich and the Language of New Media). Breslin, J.-R. (2011). Making Media: Foundations of Sound and Image Production (3rd ed.): Focal Press. Brinkmann, R. (1999). The Art and Science of Digital Compositing. San Diego: Morgan Kaufmann Academic Press. BrookeFreeman. (2011, Jul 29). Fruit Loops - Follow Your Nose (2000) [Advertisement]. Retrieved from https://youtu.be/Ew-Of0o_Hy0. Brophy, P. (1994). The Animation of Sound. Retrieved from http://www.philipbrophy.com/projects/chapters/animationofsound/chapter.html Brown, J. (2013). Audiovisual Palimpsests: Resynchronizing Silent Films with “Special” Music In D. Neumeyer (Ed.), The Oxford Handbook of Film Music Studies: Oxford University Press. Bruner, G. C. (1990). Music, Mood, and Marketing. Journal of Marketing, 54(4), 94-104. doi:10.2307/1251762 Bubb, J. (2012). Back to the future: multi-image screen narrative in a digital age. Journal of Media Practice, 13(1), 45-60. doi:10.1386/jmpr.13.1.45_1 Buckland, W. (1999). Between Science Fact and Science Fiction: Spielberg's Digital Dinosaurs, Possible Worlds, and the New Aesthetic Realism. Screen, 40(2), 177-192. Bukatman, S. (1990). Who Programs You? The Science Fiction of the Spectacle. In A. Kuhn (Ed.), Alien Zone: Cultural Theory and Contemporary Science Fiction Film (pp. 196-213). London: Verso. Burnett, R. (1977). Film/Technology/Ideology. Ciné-Tracts, 1(1), 6-14. Burnett, R. (2004). How Images Think. Cambridge: The MIT Press. Calabrese, O. (1992). Neo-Baroque: A Sign of the Times. Princeton: Princeton University Press. Calder, B. J. (2008). Kellogg on Advertising and Media. Chichester. Canemaker, J. (2001). Walt Disney's nine old men and the art of animation: Disney Editions Disney Pr, C/O Hachette Book Group USA, 53 State st, 9th Fl, Boston, MA, 02109.
109
Cardullo, B. (2007). Cinematic Illusions: Realism, Subjectivity, and the Avant-Garde. Newcastle: Cambridge Scholars Publishing. Carlson, B., & Gorman, M. (1990). Understanding Invention as a Cognitive Process: The Case of Thomas Edison and Early Motion Pictures, 1888-91. Social Studies of Science, 20(3), 387- 430. Carroll, N. (1995). Towards an Ontology of the Moving Image. Philosophy and Film, 66-88. Carroll, N. (2003). Engaging the Moving Image. New Haven: Yale University Press. Cartwright, L., & Goldfarb, B. (1992). Radiography, Cinematography and the Decline of the Lens. In J. Crary & S. Kwinter (Eds.), Incorporations (pp. 190-201). New York: Zone Books. Castello-Branco, P. (2010). Pure Sensations? From Abstract Film to Digital Images. Animation, 5(1), 25-40. doi:10.1177/1746847709356644 Caston, E. (2017). 'The First Cut is the Deepest': Excerpts from a Focus Group on Editing Music Videos, with Explanatory Historical and Theoretical Notes. Music, Sound & the Moving Image, 11(1), 99-118. doi:10.3828/msmi.2017.6 Causey, M. (1999). The Screen Test of the Double: The Uncanny Performer in the Space of Technology. Theatre Journal, 51(4), 383-394. Cavell, S. (1979). The World Viewed: Reflections on the Ontology of Film. Cambridge: Harvard University Press. Chandler, D. (2007). Semiotics: The Basics. London, UNITED KINGDOM: Taylor & Francis Group. Chion, M. (1994). Audio-Vision: Sound on Screen. Paris: Columbia University Press. Chladni, Ernst Florens Friedrich. (2002). In Leif J. Robinson , Wil Tirion , & P. Moore (Eds.), Astronomy Encyclopedia: Philip's. Chun, W. H. K., & Keenan, T. W. (Eds.). (2006). New Media, Old Media: A History and Theory Reader. New York: Routledge. CityGirls. (2020, Jul 13). City Girls - Jobs (Alternative Video) [Video file]. Retrieved from https://youtu.be/bFW_Zg2bTzs?t=2. Classic90sfan. (2018, Oct 22). San Giorgio Pasta Ad- Perfect Pasta (1992) [Advertisement]. Retrieved from https://youtu.be/o7kHA4kqrbQ. Cohen, J., & Meskin, A. (2004). On the Epistemic Value of Photographs. The Journal of Aesthetics and Art Criticism, 62(2), 197-210. Coleman, A. D. (1998). Depth of Field: Essays on Photography, Mass Media, and Lens Culture. Albuquerque: University of New Mexico Press. Collins, K. (2013). Playing with Sound : A Theory of Interacting with Sound and Music in Video Games. Cambridge, Massachusetts: The MIT Press. Combrink, L., & Marley, I. R. (2009). Practice-based research: Tracking creative creatures in a research context. Literator, 30(1), 177-206. doi:10.4102/lit.v30i1.74 Comolli, J.-L. (1986). Technique and Ideology: Camera, Perspective, Depth of Field [Parts 3 and 4]. In P. Rosen (Ed.), Narrative, Apparatus, Ideology: A Film Theory Reader (pp. 421-443). New York: Columbia University Press. Comolli, J.-L. (1996). Machines of the Visible. In T. Druckrey (Ed.), Electronic Culture: Technology and Visual Representation (pp. 108-117). New York: Aperture Press. Cook, G. (1992). Discourse of Advertising: Taylor & Francis Group. Cooke, D. (1959). The language of music: Oxford University Press Oxford Univ Pr on Demand, 2001 Evans rd, Cary, NC, 27513. Corbella, M., Windisch, A. K., Gaudreault, A., Barnier, M., & Sirois-Trahan, J.-P. (2013). Sound Synthesis, Representation and Narrative Cinema in the Transition to Sound (1926-1935). Cinémas, 24(1), 59-81. doi:10.7202/1023110ar Corbijn, A. (2005). The Work Of Director Anton Corbijn Vol 6: Umgd. Coulter, J. (2010). Electroacoustic Music with Moving Images: the art of media pairing. Organised Sound, 15(1), 26-34. doi:10.1017/S1355771809990239 Coyle, R. (2010). Drawn to Sound : Animation Film Music and Sonicity. London: Equinox Publishing Ltd. 110
Craton, L. G., & Lantos, G. P. (2011). Attitude toward the advertising music: an overlooked potential pitfall in commercials. Journal of Consumer Marketing, 28(6), 396-411. doi:10.1108/07363761111165912 Cubitt, S. (1991). Timeshift: On Video Culture. London: Routledge. Cubitt, S. (1998). Digital Aesthetics. London: SAGE Publications. Cunningham, C. (2003). The Work of Director Chris Cunningham. The Director File Series. DVD. Cunningham, C. (Writer). (2005). Rubber Johnny. In W. Films (Producer). UK: Warp Films. Currie, G. (1991). Photography, Painting and Perception. Journal of Aesthetics and Art Criticism, 49(1), 23-29. Currie, G. (1995). Image and Mind: Film, Philosophy and Cognitive Science. Cambridge: Cambridge University Press. Cytowic, R. (1993). The Man Who Tasted Shapes. Canada: G. P. Putnam's Sons. Dahl, I. (1974). Notes on Cartoon Music (1949). In M. Cooke (Ed.), The Hollywood film music reader (pp. 382): Oxford University Press Oxford Univ Pr on Demand, 2001 Evans rd, Cary, NC, 27513. Dancyger, K. (2007). The technique of film and video editing history, theory, and practice (4th ed.. ed.). Oxford. Dancyger, K. (2011). The technique of film and video editing : history, theory, and practice (5th ed.. ed.): Burlington, MA : Elsevier/Focal Press. Darley, A. (2000). Visual Digital Culture: Surface Play and Spectacle in New Media Genres: Routledge. Davis, E. (1994). Techgnosis, Magic, Memory, and the Angels of Information. In M. Dery (Ed.), Flame Wars: The Discourse of Cyberculture (pp. 29-60). Durham: Duke University Press. Dawtrey, A. (2003). Promo hotshot gets coin. Variety, 393(4), 20. Debord, G. (1995). The Society of the Spectacle (3rd ed.). New York: Zone Books. Debord, G. (1998). Comments on the Society of the Spectacle. London: Verso. Debord, G. (2003). Complete Cinematic Works. Edinburgh: AK Press. Deleuze, G. (1988). Bergsonism (H. Tomlinson, Trans.). New York: Zone Books. Deleuze, G. (1992). Mediations (M. Joughin, Trans.) Incorporations (pp. 280-294). New York: Zone Books. Deutsch, S. (2008). Aspects of synchrony in animation. Soundtrack, 1(2), 95-105. doi:10.1386/st.1.2.95/1 Devereaux, L., & Hillman, R. (Eds.). (1995). Fields of Vision: Essays in Film Studies, Visual Anthropology and Photography: University of California Press. Dickinson, K. (2007). Music Video and Synaesthetic Possibility. In R. Beebe & J. Middleton (Eds.), Medium cool : music videos from soundies to cellphones (pp. 351): Duke University Press Duke Univ Pr, Attn: Michael, Box 90660, Durham, NC, 27708. Diederichsen, D. (2011). Sound and Image Worlds in Pop Music. In D. Daniels & S. Naumann (Eds.), See This Sound : Audiovisuology: Distributed Art Pub Inc Distributed Art Pub Inc, C/O Perseus Distribution, 193 Edwards Dr, Jackson, TN, 38301. Disney, W. (2006). Silly symphonies [videorecording] / produced by Walt Disney: Walt Disney Home Entertainment. Dixon, W. W. (1995). It Looks at You: the Returned Gaze of Cinema. Albany: State University of New York Press. DN849. (2010, Oct 11). Spot - Strepsils Bestmen [Advertisement]. Retrieved from https://youtu.be/6aah-A_zJ48. Doane, M. A. (2002). The Emergence of Cinematic Time: Modernity, Contingency, the Archive. Cambridge: Harvard University Press. Doane, M. A. (2008). Scale and the Negotiation of 'Real' and 'Unreal' Space in the Cinema. NTU Studies in Language and Literature, 20, 1-38.
111
Donnelly, K. J. (2015). Experimental music video and television. In L. Mulvey, M. Hargreaves, & J. Sexton (Eds.), Experimental british television (pp. 166). Manchester New York: Manchester New York: Manchester University Press. Dotto, S. (2016). What you hear is what you (don't) see. Mario Wienerroither's Foley art in Musicless Musicvideos. Soundtrack, 9(1/2), 25-39. doi:10.1386/ts.9.1-2.25_1 Drake. (2020, Aug 13). Drake - Laugh Now Cry Later (Official Music Video) ft. Lil Durk [Video file]. Retrieved from https://youtu.be/RhxF9Qg5mOU?t=99. Durand, J. (1983). Rhétorique et image publicitaire. Australian Journal of Cultural Studies, 1(2). Retrieved from http://jacques.durand.pagesperso- orange.fr/Site/anglais/texte%202%20ang.htm Earnshaw, R. A., & Wiseman, N. (1992). An Introductory Guide to Scientific Visualization. Berlin: Springer-Verlag. Ebert, R. (2010, May 9). Why I Hate 3-D (And You Should Too). Newsweek. Eisenstein, S. (Ed.) (1988). Nonindifferent Nature: Film and the Structure of Things. Cambridge: Cambridge University Press. Eisenstein, S. M. (Ed.) (1947). The Film Sense. San Diego: Harcourt Brace and Company. Eisner, E. (2008). Art and knowledge. In A. L. K. Cole, J. Gary (Ed.), Handbook of the arts in qualitative research : perspectives, methodologies, examples, and issues: Los Angeles : Sage. Ellis, J. (1982). Visible Fictions. London: Routledge. Elsaesser, T. (2006). Early Film History and Multi-Media: An Archaeology of Possible Futures. In W. H. K. Chun & T. W. Keenan (Eds.), New Media, Old Media: a History and Theory Reader (pp. 13-26). New York: Routledge. Elsaesser, T., & Buckland, W. (2002). Realism in the Photographic and Digital Image ('Jurassic Park' and 'The Lost World') Studying Contemporary American Film: A Guide to Movie Analysis (pp. 195-219). London: Arnold. Encarnacao, J. (2013). Punk aesthetics and new folk : way down the old plank road: Lightning Source Inc Lightning Source Inc, 1246 Heil Quaker Blvd, LA Vergne, TN, 37086. Engberg-Pedersen, A., & Simons, O. (2018). Sound Figures: Between Physics and Aesthetics. Germanic Review, 93(4), 329-333. doi:10.1080/00168890.2018.1510825 Engeli, M. (2000). Digital Stories: The Poetics of Communication. Basel: Birkhäuser. Englis, B. G. (1991). Music Television and Its Influences on Consumers, Consumer Culture, and the Transmission of Consumption Messages. Advances in Consumer Research, 18(1), 111-114. Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of vision, 10(1). doi:10.1167/10.1.6 Everett, A., & Caldwell, J. T. (Eds.). (2003). New Media: Theories and Practices of Digitextuality. New York: Routledge. Fahlenbrach, K. (2005). Aesthetics and Audiovisual Metaphors in Media Perception. CLCWeb: Comparative Literature & Culture: A WWWeb Journal, 7(4), 1-6. doi:10.7771/1481-4374.1280 Fahlenbrach, K. (2008). Emotions in Sound: Audiovisual Metaphors in the Sound Design of Narrative Films. Projections: The Journal for Movies & Mind, 2(1), 85-103. Faulkner, M. (2006). VJ : audio-visual art and VJ culture (D. Fuse Ed.): Laurence King Chronicle Books Llc, C/O Hachette Book Group USA, 53 State st, 9th Fl, Boston, MA, 02109. Featherstone, M., & Burrows, R. (Eds.). (1995). Cyberspace/Cyberbodies/Cyberpunk: Cultures of Technological Embodiment. London: SAGE Publications. Feineman, N., & Reiss, S. (2000). Thirty Frames Per Second: The Visionary Art of the Music Video: Harry N Abrams. Feng, S., & Morris, J. D. (2016). Decoding Neural Responses To Emotion in Television Commercials. Journal of Advertising Research, 56(2), 193-204. doi:10.2501/JAR-2016-016 112
Ferg. (2020, Sep 25). A$AP Ferg - Dennis Rodman (Official Video) ft. Tyga [Video file]. Retrieved from https://youtu.be/yiABohCHXUQ?t=41. Fetveit, A. (2013). Medium-Specific Noise. In L. Hausken (Ed.), Thinking media aesthetics : media studies, film studies and the arts (pp. 265): Peter Lang Pub Inc, 29 Broadway, 18th Fl, New York, NY, 10006. Fielding, R. (1970). Hale’s Tours: Ultrarealism In the Pre-1910 Motion Picture. Cinema Journal, 10(1), 34-47. Finkel, L. H. (1992). The Construction of Perception. In J. Crary & S. Kwinter (Eds.), Incorporations (pp. 393-405). New York: Zone Books. Fiske, J. (1987). Television culture (Second edition.. ed.): London, England New York, New York : Routledge. Fortin, D. T. (2011). Architecture and Science-Fiction Film: Philip K. Dick and The Spectacle of Home: Ashgate Publishing. Frederick, S. (2013). Cinematic Collecting: The Continuous Discontinuity of the Still Frame in Oskar Fischinger’s Walking from Munich to Berlin. Animation, 8(3), 235-251. doi:10.1177/1746847713501829 Frierson, M. (2017). Film and video editing theory: how editing creates meaning. Frigerio, A., Giordani, A., & Mari, L. (2013). On Representing Information: A Characterization of the Analog/Digital Distinction. Dialectica: International Journal of Philosophy & Official Organ of the ESAP, 67(4), 455-483. doi:10.1111/1746-8361.12040 Frith, S. (1986). Art versus technology: the strange case of popular music. Media, Culture & Society, 8(3), 263-279. doi:10.1177/016344386008003002 Fujiki, H. (2006). Benshi as Stars: The Irony of the Popularity and Respectability of Voice Performers in Japanese Cinema. Cinema Journal, 45(2), 68-84. doi:10.1353/cj.2006.0016 Furniss, M. (2017). Art in Motion, Revised Edition : Animation Aesthetics (Vol. Revised edition). Herts: John Libbey Publishing. Future. (2020, Jul 20). Future - Posted With Demons (Official Music Video) [Video file]. Retrieved from https://youtu.be/Z7DdyBt09P8. Gabriel, C. (1978). An Experimental Study of Deryck Cooke's Theory of Music and Meaning. Psychology of Music, 6(1), 13-20. doi:10.1177/030573567861002 Gallagher, M. (2009). The Music Tech Dictionary : A Glossary of Audio-related Terms and Technologies. Boston, MA: Course PTR. gamelovercommercials. (2013, May 16). 1992 All detergent commercial [Advertisement]. Retrieved from https://www.youtube.com/watch?v=zpYDDFjgpeo. Gardner, M. (1985). Mood States and Consumer Behavior: A Critical Review. Journal of Consumer Research, 12(3), 281. doi:10.1086/208516 Garwood, I. (2012). Roto-Synchresis: Relationships between Body and Voice in Rotoshop Animation. Animation, 7(1), 39-57. doi:10.1177/1746847711428850 Gaut, B. (2009). Digital Cinema. In P. Livingston & C. Plantinga (Eds.), The Routledge Companion to Philosophy and Film. Abingdon: Routledge. Geduld, H. M. (1975). The birth of the talkies : from Edison to Jolson: Indiana University Press Indiana Univ Pr, Attn Pat Hoefling, 601 N Morton st, Bloomington, IN, 47404-3797. Gerow, A. (2010). Visions of Japanese Modernity : Articulations of Cinema, Nation, and Spectatorship, 1895-1925. Berkeley: University of California Press. Gibson, R. (2010). The Known World. Sydney College of the Arts at the University of Sydney, 12. Gibson, R. (2018). Foreword: Cognitive Two-Steps. In C. Batty & S. Kerrigan (Eds.), Screen Production Research : Creative Practice As a Mode of Enquiry: Springer Verlag Springer Verlag, 1120 Rte 22 East, Bridgewater, NJ, 08807. Glassner, A. (2004). Interactive Storytelling: Techniques for 21st Century Fiction. Natick: AK Peters, Ltd.
113
Goetz, I., Urbaschek, S., & Weibel, P. (Eds.). (2003). Fast Forward: Media Art: Sammlung Goetz. Karlsruhe: Ingvild Goetz and Stephan Urbaschek. Goldmark, D. I. (2005). Tunes for 'Toons : Music and the Hollywood Cartoon. Berkeley: University of California Press. Gondry, M. (Writer). (2001). The Hardest Button To Button. Gordon, W. T. (2010). McLuhan : a guide for the perplexed: Continuum Bloomsbury USA Academic, C/O Mps, 16365 James Madison Hwy, Gordonsville, VA, 22942. Grau, O. (2003). Virtual Art: From Illusion to Immersion (revised ed.). Cambridge: The MIT Press. Greenfield, P., Farrar, D., & Beagles-Roos, J. (1986). Is the medium the message?: An experimental comparison of the effects of radio and television on imagination. Journal of Applied Developmental Psychology, 7(3), 201-218. doi:10.1016/0193-3973(86)90029-8 Greenfield, P. M., Yut, E., Chung, M., Land, D., Kreider, H., Pantoja, M., & Horsley, K. (1990). The Program-Length Commercial: A Study of the Effects of Television/Toy Tie-Ins on Imaginative Play. Psychology & Marketing, 7(4), 237-255. doi:10.1002/mar.4220070402 Griffith, D. W. (1916). Intolerance [videorecording] / Wark Producing Corporation; written and directed by D. W. Griffith: Kino on Video. Grist, L. (2009). Whither Realism? Bazin Reconsidered. In L. Nagib & C. Mello (Eds.), Realism and the Audiovisual Media (pp. 20-30). Hampshire: Palgrave Macmillan. Gunna. (2019, Jul 9). Gunna - Baby Birkin (Starring Jordyn Woods) [Official Video] [Video file]. Retrieved from https://youtu.be/3BOuz4dBkkE?t=38. Gunna. (2019, May 7). Gunna - Three Headed Snake ft. Young Thug [Official Video] [Video file]. Retrieved from https://youtu.be/0kbOaQOxiQU?t=13. Gunning, T. (1994). The World as Object Lesson: Cinema Audiences, Visual Culture and the St. Louis World´s Fair, 1904. Film History, 6(4), 422-444. Gunning, T. (1995a). "Animated pictures", tales of cinema's forgotten future. Michigan Quarterly Review, 34(4), 464. Gunning, T. (1995b). Phantom Images and Modern Manifestations: Spirit Photography, Magic Theater, Trick Films, and Photography´s Uncanny. In P. Petro (Ed.), Fugitive Images: From Photography to Video (pp. 42-71). Bloomington: Indiana University Press. Gunning, T. (2004). What's the Point of an Index? Or, Faking Photographs. NORDICOM Review, 5(1/2), 39-49. Gunning, T. (2007). Moving Away from the Index: Cinema and the Impression of Reality. Differences: A Journal of Feminist Cultural Studies, 18(1), 29-52. Gupta, P. B., & Lord, K. R. (1998). Product Placement in Movies: The Effect of Prominence and Mode on Audience Recall. Journal of Current Issues & Research in Advertising, 20(1), 47-59. doi:10.1080/10641734.1998.10505076 Gurevitch, L. (2010). The Cinemas of Interactions: Cinematics and the ‘Game Effect’ in the Age of Digital Attractions. Senses of Cinema(57). Guter, E. (2010). Aesthetics A-Z. Edinburgh: Edinburgh University Press. Hagener, M. (2008). The Aesthetics of Displays: How the Split Screen Remediates Other Media. The Refractory: a Journal of Entertainment Media, 14. Hagood, M. (2014). Unpacking a Punch: Transduction and the Sound of Combat Foley in Fight Club. Cinema Journal, 53(4), 98-120. Hall, D., & Fifer, S. J. (Eds.). (1990). Illuminating Video: An Essential Guide to Video Art. San Francisco: Aperture Press in association with The Bay Area Video Coalition. Halliday, S. (2013). Sonic Modernity : Representing Sound in Literature, Culture and the Arts. Edinburgh: Edinburgh University Press. Hanney, R. (2013). Towards a situated media practice: Reflections on the implementation of project- led problem-based learning. Journal of Media Practice, Volume 14(Number 1). Hansen, M. (1991). Babel and Babylon: Spectatorship in American Silent Film. Cambridge: Harvard University Press.
114
Harley, R. (1996). That's Interaction! Audience Participation in Entertainment Monopolies. Convergence: The Journal of Research into New Media Technologies, 2(2), 103-123. Harley, R. (Ed.) (1993). New Media Technologies. Sydney: Australian Film Television and Radio School. Harpole, C., H. (1980). Ideological and Technological Determinism in Deep-Space Cinema Images: Issues in Ideology, Technological History, and Aesthetics. Film Quarterly, 33(3), 11-22. Harris, R. J., Sturm, R. E., Klassen, M. L., & Bechtold, J. I. (1986). Language in Advertising a Psycholinguistic Approach. Current Issues and Research in Advertising, 9(1-2), 1-26. doi:10.1080/01633392.1986.10505389 Hartmann, W. M. (2013). Principles of Musical Acoustics (1st ed. 2013.. ed.): New York, NY : Springer New York : Imprint: Springer. Hassan, R., & Thomas, J. (Eds.). (2006). The New Media Theory Reader. Maidenhead: Open University Press. Hatfield, J. (Ed.) (2006). Experimental Film and Video: An Anthology. Eastleigh: John Libbey Publishing. Hausken, L. (2013a). Thinking Media Aesthetics : Media Studies, Film Studies and the Arts. Frankfurt: Peter Lang AG. Hausken, L. (2013b). Thinking media aesthetics : media studies, film studies and the arts: Peter Lang Pub Inc, 29 Broadway, 18th Fl, New York, NY, 10006. Helmreich, S. (2015). Transduction. In D. Novak & M. Sakakeeny (Eds.), Keywords in sound (pp. 260). Duke Univ Pr: Duke Univ Pr, Attn: Michael, Box 90660, Durham, NC, 27708. Herzog, A. (2007). Illustrating Music: The Impossible Embodiments of the Jukebox Film. In R. Beebe & J. Middleton (Eds.), Medium cool : music videos from soundies to cellphones (pp. 351): Duke University Press. Hodgins, J. K., Obrien, J. F., & Bodenheimer, J. R. E. (1999). COMPUTER ANIMATION. Encyclopedia of Electrical & Electronics Engineering, 3, 686-690. Hood, J. L. (1999). Audio Electronics (Vol. 2nd ed). Oxford [England]: Newnes. Hyde, J. (2012). Musique Concrete Thinking in Visual Music Practice: Audiovisual silence and noise, reduced listening and visual suspension. Organised Sound, 17(2), 170-178. doi:http://dx.doi.org/10.1017/S1355771812000106 IllPropaganda. (2012, Mar 8). Coldcut - Timber [Video file]. Retrieved from https://youtu.be/5- wl7Xk5FoY. Isaacs, B. (2007). The Cinematic Real: Aesthetics and Spectacle. Sydney Studies in English, 33, 96- 124. Jacobs, L. (2014). Film Rhythm After Sound : Technology, Music, and Performance. Oakland, California: University of California Press. Jameson, F. (1991). Postmodernism, or, The Cultural Logic of Late Capitalism. Durham: Duke University Press. Jhally, S. (2002). The Anthropology of media : a reader. In K. M. Askew & R. R. Wilk (Eds.), The Anthropology of media : a reader (pp. 416). John Wiley & Sons Inc, 1120 Rte 22 East, Bridgewater, NJ, 08807: Blackwell Publishers. JimmyEatWorld. (2010, Jul 8). Jimmy Eat World - The Middle (Official Music Video) [Video file]. Retrieved from https://youtu.be/oKsxPW6i3pM. Johnson, J. (1995). Perception. In G. Rzevski (Ed.), Mechatronics: Designing Intelligent Machines (Vol. 1, Perception, Cognition and Execution, pp. 101-158). Oxford: Butterworth Heinemann. Jones, R., & Nevile, B. (2005). Creating Visual Music in Jitter: Approaches and Techniques. Computer Music Journal, 29(4), 55-70. doi:10.1162/014892605775179900 Jones, S. (1988). Cohesive but not coherent: music videos, narrative and culture. Popular Music and Society, 12(Winter, 1988, No. 4). Josephson, S. G. (2015). From idolatry to advertising : visual art and contemporary culture: London, England New York, New York : Routledge. 115
Jueneman, F. (1998). Chladni's legacy. R&D Magazine, 40(7), 11. Kalinak, K. M. (2010). Film Music: A Very Short Introduction. Oxford: Oxford University Press. Kane, B. (2014). Pierre Schaeffer, the sound object and the acousmatic reduction Sound unseen : acousmatic sound in theory and practice (pp. 318): Oxford Univ Pr Oxford Univ Pr, 2001 Evans rd, Cary, NC, 27513. KanyeWest. (2009, 17 Jun). Kanye West - Gold Digger ft. Jamie Foxx [Video file]. Retrieved from https://youtu.be/6vwNcNOTVzY?t=20. Kaplan, E. A. (1987). Rocking around the clock : music television, postmodernism, and consumer culture: Methuen Taylor & Francis, C/O Kentucky Dist Center, 7625 Empire Dr, Florence, KY, 41042. Keating, A. (2013). Video-making, Harlem Shaking: Theorizing the interactive amateur. New Cinemas: Journal of Contemporary Film, 11(2/3), 99-110. doi:10.1386/ncin.11.2-3.99_1 Keazor, H., & Wubbena, T. (2010). Rewind, Play, Fast Forward : The Past, Present and Future of the Music Video: Columbia Univ Pr Columbia Univ Pr, C/O Perseus Distribution, 210 American Dr, Jackson, TN, 38301. Kendall, G., Haworth, C., & Cádiz, R. (2014). Sound Synthesis with Auditory Distortion Products. Computer Music Journal, 38(4), 5-23. doi:10.1162/COMJ_a_00265 Kerner, M. M. (1989). The art of the sound effects editor: Focal Press Taylor & Francis, C/O Kentucky Dist Center, 7625 Empire Dr, Florence, KY, 41042. Kerrigan, S., Leahy, G., & Cohen, H. (2016). Still a burning issue: measuring screen production research. Studies in Australasian Cinema, 10(1), 79-96. doi:10.1080/17503175.2016.1140257 Kessler, F. (2009). What You Get is What You See: Digital Images and the Claim on the Real. In M. Van den Boomen, S. Lammes, A.-S. Lehmann, J. Raessens, & M. T. Schäfer (Eds.), Digital Material: Tracing New Media in Everyday Life and Technology. Amsterdam: Amsterdam University Press. Kilyeni, A. (2012). The Promise of Instant Beauty in the Language of Print Advertisements for Cosmetics. Scientific Bulletin of the Politehnica University of Timisoara. Transactions on Modern Languages / Buletinul Stiintific al Universitatii Politehnica din Timisoara. Seria Limbi Moderne, 11(1/2), 19-30. KimCarnesVEVO. (2009, Feb 28). Kim Carnes - Bette Davis Eyes [Video file]. Retrieved from https://youtu.be/EPOIS5taqA8. Kinder, M. (1984). Music video and the spectator: television, ideology and dream. Film Quarterly, 38, 2. Korten, D. C. (2015). When corporations rule the world: Random House Inc Random House Inc, Attn Order Entry, 400 Hahn rd, Westminster, MD, 21157. Krauss, R. E. (1990). A Note on Photography and the Simulacral. In C. Squires (Ed.), The Critical Image: Essays on Contemporary Photography (pp. 15-27). Seattle: Bay Press. Kress, G. R., & Van Leeuwen, T. (2001). Multimodal discourse : the modes and media of contemporary communication: Arnold Quercus, C/O Hachette Book Group USA, 53 State st, 9th Fl, Boston, MA, 02109. Krishna, A. (2012). An integrative review of sensory marketing: Engaging the senses to affect perception, judgment and behavior. Journal of Consumer Psychology, 22(3), 332-351. doi:10.1016/j.jcps.2011.08.003 Kroon, R. W. (2014). A/V a to z: An encyclopedic dictionary of media, entertainment and other Audiovisual terms. Jefferson, NC: McFarland. Kunkel, D. L. (1984). Children's understanding of television advertising: The impact of host-selling: University of Southern California Digital Library (USC.DL). Lapedis, H. (1999). Popping the question: the function and effect of popular music in cinema. Pop. Mus., 18(3), 367-379. doi:10.1017/S0261143000008928 116
Leggott, J. (2016). Come to Daddy? Claiming Chris Cunningham for British Art Cinema. Journal of British Cinema & Television, 13(2), 243-261. doi:10.3366/jbctv.2016.0311 Leonard, K. (2016). Using Resources For Silent Film Music. Fontes Artis Musicae, 63(4), 259-276. LerchVid. (2018, Mar 18). 90's Toy Commercials [Advertisement]. Retrieved from https://youtu.be/agHW_y6sgDk. Lindberg, C. A., & Stevenson, A. (2011). New Oxford American dictionary (3rd ed. / edited by Angus Stevenson, Christine A. Lindberg.. ed.): Oxford : Oxford University Press. LinkinPark. (2009, Oct 24). Crawling (Official HD Video) - Linkin Park [Video file]. Retrieved from https://youtu.be/Gd9OhYroLN0. Lowe, M. L., & Haws, K. L. (2017). Sounds Big: The Effects of Acoustic Pitch on Product Perceptions. Journal of Marketing Research, 54(2), 331-346. doi:http://www.marketingpower.com/jmr Lye, L. (Writer). (1936). Rainbow Dance. Lyndon, C. S. W., & Simon, M. (2017). Music As Multimodal Discourse : Semiotics, Power and Protest. New York: Bloomsbury Academic. MacRury, I. (1998). Advertising (1st edition. ed.): London : Taylor & Francis Group. Majoe, D., Kulka, I., & Schacher, J. (2009). Towards an interactive multimedia experience for club music and dance (pp. 95-104). Manovich, L. (2001). The Language of New Media. Cambridge: The MIT Press. Marchand, R. (1985). Advertising the American Dream: Making Way for Modernity, 1920-1940: University of California Press. MarioWienerroither. (2014, Jul 8). Musicless Musicvideo / ELVIS PRESLEY - Blue Suede Shoes [Video file]. Retrieved from https://youtu.be/5Jd9AmepgdM. Marks, L. (1989). ON CROSS-MODAL SIMILARITY - THE PERCEPTUAL STRUCTURE OF PITCH, LOUDNESS, AND BRIGHTNESS. J. Exp. Psychol.-Hum. Percept. Perform., 15(3), 586-602. Marks, L. E. (1982). Synesthetic perception and poetic metaphor. Journal of Experimental Psychology: Human Perception and Performance, 8(1), 15-23. doi:10.1037/0096-1523.8.1.15 Marks, L. E. (1987). On Cross-Modal Similarity: Auditory-Visual Interactions in Speeded Discrimination. Journal of Experimental Psychology: Human Perception and Performance, 13(3), 384-394. doi:10.1037/0096-1523.13.3.384 massbetelnut. (2012, Nov 12). The Beach Boys Shred I Get Around [Advertisement]. Retrieved from https://youtu.be/w-0CS-T1HUQ. Mc, S. (2014, Sep 22). Meshach Taylor 1987 Surf Detergent Commercial [Advertisement]. Retrieved from https://youtu.be/r2jwvPbwqkc. McCracken, G. (1986). Culture and Consumption: A Theoretical Account of the Structure: A Theoretical Account of the Structure and Movement of the Cultural Meaning of Consumer Goods. Journal of Consumer Research, 13(1), 71-84. doi:10.1086/209048 Mclaren, N. (Writer). (1949). A Phantasy in Colors. McLuhan, M. (1964). Understanding Media: the Extensions of Man (1st MIT Press ed.). Cambridge: The MIT Press. Melara, R. D. (1989). Dimensional interaction between color and pitch. Journal of Experimental Psychology: Human Perception and Performance, 15(1), 69-79. doi:10.1037/0096- 1523.15.1.69 Meline, C. W. (1976). Does the Medium Matter? Journal of Communication, 26(3), 81-89. doi:10.1111/j.1460-2466.1976.tb01908.x Meyers, D. (Writer). (2004). Jay-Z, Dirt Off Your Shoulder. YouTube. Mollaghan, A. (2015). Introduction. In A. Mollaghan (Ed.), The Visual Music Film (pp. 1-8). London: Palgrave Macmillan UK. MoneyBaggYo. (2020, Jul 17). Moneybagg Yo - Protect Da Brand (feat. DaBaby) (Official Music Video) [Video file]. Retrieved from https://youtu.be/PHqN_919ACA?t=39. Morgan, D. (2006). Rethinking Bazin: Ontology and Realist Aesthetics. Critical Inquiry, 32(3), 443-481. mrbriefcaseMUSIC. (2016, Jun 7). The Arrows - Talk Talk [Video file]. Retrieved from https://youtu.be/YC_wqQxKW2U.
117
MrStephenRJohnson. (2010, Nov 17). Stephen R. Johnson and Peter Gabriel Interview On Their Videos Together [Video file]. Retrieved from https://youtu.be/J55GrBQtHPk. mycommercials. (2007, Oct 6). Hair Loss Commercial (1990) [Advertisement]. Retrieved from https://youtu.be/ferbU_zJix8. Nanayakkara, S. C., Wyse, L., Ong, S. H., & Taylor, E. A. (2013). Enhancing Musical Experience for the Hearing-Impaired Using Visual and Haptic Displays. Human-Computer Interaction, 28(2), 115-160. doi:10.1080/07370024.2012.697006 Nelson, R. (2013). Practice as Research in the Arts Principles, Protocols, Pedagogies, Resistances: London : Palgrave Macmillan UK Palgrave Macmillan. Neumeyer, J. B., David. (2014). Music And The Ontology Of The Sound Film: The Classical Hollywood System The Oxford Handbook of Film Music Studies. Oxford: Oxford University Press. Newman, M. Z. (2014). Video revolutions : on the history of a medium: Columbia Univ Pr Columbia Univ Pr, C/O Perseus Distribution, 210 American Dr, Jackson, TN, 38301. NoDoubtTV. (2009, Oct 7). No Doubt - Hey Baby [Video file]. Retrieved from https://youtu.be/wt1YkGO2Ieo. northbaysports. (2019, May 20). All (1980s) - Laundry Detergent Commercial [Advertisement]. Retrieved from https://youtu.be/IdqATwzcgfM. O'Brien, C. (2005). Cinema's Conversion to Sound : Technology and Film Style in France and the U.S. Bloomington: Indiana University Press. O'Brien, J. F., & Hodgins, J. K. (2000). Animating Fracture. Communications of the ACM, 43(7), 68-75. doi:10.1145/341852.341865 Olsen, G. (1994). Observations: The Sounds of Silence: Functions and Use of Silence in Television Advertising. Journal of Advertising Research, 34(5), 89. Owen, A. N. N. (2011). Neuroanimatics: Hyperrealism and digital animation's accidental mark. Animation Practice, Process & Production, 1(2), 315-345. doi:10.1386/ap3.1.2.315_1 OwlCity. (2009, Dec 15). Owl City - Fireflies (Official Video) [Video file]. Retrieved from https://youtu.be/psuRGfAaju4?t=13 oztvheritage. (2012, Oct 12). Victoria Bitter TV ads 1968-1996 [Advertisement]. Retrieved from https://youtu.be/nszIJDswTBM. Parise, C., & Spence, C. (2008). Synesthetic congruency modulates the temporal ventriloquism effect. Neuroscience Letters, 442(3), 257-261. doi:10.1016/j.neulet.2008.07.010 Partington, A. (2008). The best bits: Non-narrative pleasures and creative practice. Journal of Media Practice, 9(1), 9-18. doi:10.1386/jmpr.9.1.9_1 Partyka, J. (1999). Analog vs. Digital: no clear victor. EMedia, 12(12), 50. Pesic, P. (2006). Isaac Newton and the mystery of the major sixth: a transcription of his manuscript 'Of Musick' with commentary. Interdisciplinary Science Reviews, 31(4), 291-306. doi:10.1179/030801806X143268 PeterGabriel. (2012, May 4). Peter Gabriel - Big Time [Video file]. Retrieved from https://youtu.be/PBAl9cchQac. PeterGabriel. (2012, Sep 42). Peter Gabriel - Sledgehammer (HD version) [Video file]. Retrieved from https://youtu.be/OJWJE0x7T4Q. Peverini, P. (2010). The Aesthetics of Music Videos: An Open Debate Rewind, Play, Fast-Forward. The Past, Present and Future of the Music Video: Transaction Publishers. PhilCollins. (2010, May 11). Phil Collins - Take Me Home (Official Music Video) [Video file]. Retrieved from https://youtu.be/sRY1NG1P_kw?t=266. Pollay, R. W. (1986). The Distorted Mirror: Reflections on the Unintended Consequences of Advertising. Journal of Marketing, 50(2), 18-36. Powell, J. (2011). How music works : the science and psychology of beautiful sounds, from Beethoven to the Beatles and beyond: Little, Brown Little Brown & Co, 53 State st, 9th Fl, Boston, MA, 02109. 118
Preston, I. L. (1996). The great American blow-up : puffery in advertising and selling (Rev. edition. ed.). Madison, Wis. QueenOfficial. (2013, Oct 16). Queen - The Show Must Go On (Official Video) [Video file]. Retrieved from https://youtu.be/t99KH0TR-J4?t=12. Rahn, J. (2008). Digital content: video as research. In A. L. K. Cole, J. Gary (Ed.), Handbook of the arts in qualitative research : perspectives, methodologies, examples, and issues: Los Angeles : Sage. Records, W. (2005). Rubber Johnny: Chris Cunningham. Retrieved from http://mstation.org/chris_cunningham_rubber_j.php remhq. (2009, Oct 26). R.E.M. - Shiny Happy People (Official Music Video) [Video file]. Retrieved from https://youtu.be/YYOKMUTTDdA?t=35. Ren, Z., Yeh, H., & Lin, M. C. (2010). Synthesizing contact sounds between textured models (pp. 139- 146). Ribas, L. (2016). Sound and image relations: a history of convergence and divergence. Rodowick, D. N. (2007). The Virtual Life of Film. Cambridge: Harvard University Press. Rose, G. (2016). Visual methodologies : an introduction to researching with visual materials: Sage Pubns Sage Pubns, 2455 Teller rd, Thousand Oaks, CA, 91320. Røssaak, E. (2012). Between Stillness and Motion: Film, Photography, Algorithms: Amsterdam University Press. Russell, Cristel A. (2002). Investigating the Effectiveness of Product Placements in Television Shows: The Role of Modality and Plot Connection Congruence on Brand Memory and Attitude. Journal of Consumer Research, 29(3), 306-318. doi:10.1086/344432 Ruttman, W. (Writer). (1921). Lichtspiel Opus I (1921). Sahlins, M. (1972). Stone age economics. Chicago: Aldine-Atherton. Sánchez-Olmos, C., & Viñuela, E. (2017). The Musicless Music Video as a Spreadable Meme Video: Format, User Interaction, and Meaning on YouTube. International Journal of Communication (19328036), 11, 3634-3654. Schaefer, S. C. (2017). Illuminating the Divine: The Magic Lantern and Religious Pedagogy in the USA, ca. 1870–1920. Material Religion, 13(3), 275-300. doi:10.1080/17432200.2017.1308176 Schaeffer, P. (1966). Treatise on musical objects : essays across disciplines: Univ of California Pr Univ of California Pr, C/O Ips Jackson, 210 American Dr, Jackson, TN, 38301. Schifrin, L., & Feist, J. (2011). Music composition for film and television: Berklee Hal Leonard Corp, Attn: Joel Aldinger, Po Box 13819, 7777 W Bluemound rd, Milwaukee, WI, 53213. Schudson, M. (1993). Advertising, the uneasy persuasion its dubious impact on American society. Oxfordshire, England New York. Schwartz, B. (1967). The Social Psychology of the Gift. American Journal of Sociology, 73(1), 1-11. doi:10.1086/224432 Scott, L. M. (1990). Understanding Jingles and Needledrop: A Rhetorical Approach to Music in Advertising. Journal of Consumer Research, 17(2), 223-236. SisqoVEVO. (2009, July 17). Sisqo - Thong Song [Video file]. Retrieved from https://youtu.be/Oai1V7kaFBk. software. (2011, Oct 27). Oneohtrix Point Never - Sleep Dealer [OFFICIAL VIDEO] [Video file]. Retrieved from https://youtu.be/3r3cBP1xgag. Sonnenschein, D. (2001). Sound design : the expressive power of music, voice and sound effects in cinema: Studio City, Calif. : Michael Wiese. Spadoni, R. (2007). Uncanny Bodies : The Coming of Sound Film and the Origins of the Horror Genre. Berkeley: University of California Press.
119
Spence, C. (2012). Managing sensory expectations concerning products and brands: Capitalizing on the potential of sound and shape symbolism. Journal of Consumer Psychology, 22(1), 37-54. doi:10.1016/j.jcps.2011.09.004 Starr, D. (1985, 1985/04/26/). Music that helps people see better. Back Stage, 26, 44B. StephenPayne. (2018, Aug 24). Saturday Morning Commercials from 1980-1989 [Advertisement]. Retrieved from https://youtu.be/Rxm13DpWGf8. Stout, P. A., & Leckenby, J. D. (1986). Measuring Emotional Response to Advertising. Journal of Advertising, 15(4), 35-42. doi:10.1080/00913367.1986.10673036 . Strepsils 'Bestmen' Television Advertisement. www.youtube.com. SubPop. (2009, Feb 11). Fleet Foxes - Mykonos [OFFICIAL VIDEO] [Video file]. Retrieved from https://youtu.be/DT-dxG4WWf4?t=42 Taberham, P. (2018). A General Aesthetics of American Animation Sound Design. Animation, 13(2), 131-147. doi:10.1177/1746847718782889 takutaq. (2008, Mar 22). 1991 Plymouth Voyager Commercial [Advertisement]. Retrieved from https://youtu.be/0b939O7dGqQ. Taylor, P. G. (2007). Press Pause: Critically Contextualizing Music Video in Visual Culture and Art Education. Studies in Art Education: A Journal of Issues and Research in Art Education, 48(3), 230-246. TaylorSwift. (2019, Jun 17). Taylor Swift - You Need To Calm Down [Video file]. Retrieved from https://youtu.be/Dkk9gvTmCXY?t=46. Tetzlaff, D. J. (1986). MTV and the Politics of Postmodern Pop. Journal of Communication Inquiry, 10(1), 80-91. doi:10.1177/019685998601000111 TravisScott. (2017, July 13). Travis Scott - BUTTERFLY EFFECT (Official Music Video) [Video file]. Retrieved from https://youtu.be/_EyZUTDAH0U?t=48. Treharne, E., & Walker, G. (2017). Textual Distortion. Melton, UNITED KINGDOM: Boydell & Brewer, Limited. UKHaribo. (2018, 26 July). HARIBO Fruitilicious Advert 2018 - LIft [Advertisement]. Retrieved from https://youtu.be/zhqv6hf7khY. van Leeuwen, T. (2012). The critical analysis of musical discourse. Critical Discourse Studies, 9(4), 319-328. doi:10.1080/17405904.2012.713204 van Niekerk, A., & Conradie, M. (2016). Branding through art: the commercial value of visual and linguistic signs of art. Critical Arts: A South-North Journal of Cultural & Media Studies, 30(2), 233-251. doi:10.1080/02560046.2016.1187795 Vernallis, C. (2004). Experiencing Music Video : Aesthetics and Cultural Context: New York : Columbia University Press. Vernallis, C. (2008). Strange People, Weird Objects: The Nature of Narrativity, Character, and Editing in Music Videos. In R. Beebe & J. Middleton (Eds.), Medium cool : music videos from soundies to cellphones. Durham, NC. Walker, P., Bremner, J. G., Mason, U., Spring, J., Mattock, K., Slater, A., & Johnson, S. P. (2010). Preverbal Infants’ Sensitivity to Synaesthetic Cross-Modality Correspondences. Psychological Science (Sage Publications Inc.), 21(1), 21-25. doi:10.1177/0956797609354734 Wallis, T. a. P., M. (2008). Film: A Critical Introduction (Second ed.). London: Laurence King Publishing. Warner, T. (2017). Narrating Sound: The Pop Video in the Age of the Sampler. In R. Stilwell & P. Powrie (Eds.), Changing Tunes : The Use of Pre-existing Music in Film. Taylor & Francis, C/O Kentucky Dist Center, 7625 Empire Dr, Florence, KY, 41042: Taylor & Francis. Wells, P. (1998). Understanding Animation. London, UNITED KINGDOM: Routledge. White, B. (2006). "As if they didn't hear the music," Or: How I Learned to Stop Worrying and Love Mickey Mouse. Opera Quarterly, 22(1), 65-89. doi:10.1093/oq/kbi108 Williamson, J. (1978). Decoding Advertisements. London, UNITED STATES: Marion Boyars.
120
Wolfson, D. (2003). Annotation. Theories of Media, Winter 2003. Retrieved from http://csmt.uchicago.edu/annotations/CHION.HTM#:~:text=Chion%20cites%20%22added%2 0value%22%20as,the%20sum%20of%20its%20parts. Wood, R. (2001). Toward an Ontology of Film: A Phenomenological Approach. Salon, 5(24). Wright, B. (2014). Footsteps with character: the art and craft of Foley (Vol. 55, pp. 204-220): Oxford University Press / USA. Zavrl, N. (2015). Spectatorship and Synchronous Sound Before the Transition: A Contextual Analysis of Chronophone, Phonofilm, and Movietone Shorts. Film Matters, 6(3), 34-41. doi:10.1386/fm.6.3.34_1 ТехноМикс90. (2016, Aug 2). Ian Van Dahl - Castles In The Sky 2000 [Video file]. Retrieved from https://youtu.be/0aa7KWLkzXc. ЮмарвинаСмешинка100%. (2019, 12 May). Terminator 2 Judgment Day 1991 & Terminator Salvation 2009 Toy Commercial Collection [Advertisement]. Retrieved from https://youtu.be/udk-eZHcrZY. 00
121