The Effect of Audio Cues and Sound Source Stimuli on Looming
Total Page:16
File Type:pdf, Size:1020Kb
The Effect of Audio Cues and Sound Source Stimuli on Looming Perception Sonia Wilkie Submitted in partial fulfilment of the requirements of the Degree of Doctor of Philosophy School of Electronic Engineering and Computer Science Queen Mary, University of London January 28, 2015 Statement of Originality I, Sonia Wilkie, confirm that the research included within this thesis is my own work or that where it has been carried out in collaboration with, or supported by others, that this is duly acknowledged below and my contribution indicated. Previously published material is also acknowledged below. I attest that I have exercised reasonable care to ensure that the work is original, and does not to the best of my knowledge break any UK law, infringe any third partys copyright or other Intellectual Property Right, or contain any confidential material. I accept that the College has the right to use plagiarism detection software to check the electronic version of the thesis. I confirm that this thesis has not been previously submitted for the award of a degree by this or any other university. The copyright of this thesis rests with the author and no quotation from it or in- formation derived from it may be published without the prior written consent of the author. Signature: Date: Abstract Objects that move in depth (looming) are ubiquitous in the real and virtual worlds. How humans interact and respond to these approaching objects may affect their continued survival in both the real and virtual words, and is dependent on the individual's capacity to accurately interpret depth and movement cues. In computer-generated environments, including hyper and virtual reality, film, and gaming, these cues are often complex sounds with multiple audio cues that are creatively designed for maximum effect. To accurately generate a dynamic and rich perception of looming objects, the design of such complex stimuli should be based on a firm scientific foundation that encompasses what we know about how people visually and aurally perceive events and interac- tions. Conversely, many psychological studies investigating auditory looming depict the ob- ject's movement using simple audio cues, such as an increase in the amplitude, which are applied to tones that are not regularly encountered in the natural world, such as sine, triangle, or square waves. Whilst the results from these studies have provided im- portant information on human perception and responses, technological advances now allow us to present complex audiovisual stimuli and to collect measurements on human perception and responses to real and hyper-real stimuli. The research in this thesis begins to address the gap that exists between the research corpus and industry usage. This is initially accomplished by conducting a feature analysis of the audio cues and complex sounds constructed by sound designers for film scenes presenting objects moving in depth. This is followed by a perceptual study measuring human responses, both physical and emotional, to the complex audio cues designed for the film scenes. Using physical models, we then select a number of audio cues for closer inspection and introduce the parameter of `room reflections’ as an audio cue. We investigate whether or not human responses to various audio cues differ when they are presented individually or in combination, or when they are applied to an artificial (square wave) sound source or a real world sound source. Finally, we test the capacity of these audio cues to bias multimodal auditory-visual perception of an approaching object. For Murdoch, Netta, and Peggy Acknowledgements First and foremost, I would like to thank my supervisor Dr. Tony Stockman for his years of sound advice, his cheerful disposition, for providing me with a great deal of freedom to explore the topics of my choice and work on the research areas that interest me the most. Thank you to my examiners Dr. Peter Lennox and Prof. Peter McOwen. Your rig- orous scrutiny of the thesis, grilling at the viva, and advice has been greatly appreci- ated. Thank you to my thesis readers, Brandy Monteith & Brenton Wilkie, Greg Wilkie, Dogu Erdener, Alice Clifford, Nerissa Haby, Farzana Rita, and Prof. John Neuhoff. To the past and present researchers of the Centre for Digital Music and the Cognitive Science research groups, you have made these four years easily my most pleasant re- search experience. It has been a privilege to work along side of you. Special thank you to Mark Plumbley, Marcus Pearce, Andrea Halpern, Josh Reiss, Anssi Klapuri, Mark Sandler, Simon Dixon, Elaine Chew, Panos Kudumakis, Andrew McPherson, Al- ice Clifford, Emmanouil Benetos, Michael Terrell, Georgy Fazekas, Peter Foster, Andy Simpson, Daniele Barchiesi, Mathieu Barthet, Magdalena Chudy, Joachim Ganseman, Steven Hargreaves, Robert Tubb, Boris Mailhe, Martin Morrell, Ken OHanlon, Thomas Wilmering, Adam Stark, Matthias Mauch, Asterios Zacharakis, Holger Kirchhoff, Dan Stowell, Luis Figueira, Andrew Robertson, Chunyang Song, Yading Song, Becky Stew- art, Andrew Nesbit, Ireti Olowe, Oussama Metatla, Mi Tian, Tian Cheng, Bob Sturm, David Ronan, Joachim Ganseman, Duncan Menzies, Jordan Smith, Saoirse Finn, Pol- lie Barden, Ioana Dalca, Nela Brown, Doon McDonald, Sebastian Ewert, Chris Harte. The support staff Melissa Yeo, Sharon Cording, Hayley Cork, Colin Powell, Ho Huen, Nicole Gozna, Joan Hunter, Carly Wheeler, Tracy Bussoli, Stephen Hutchinson. Thank you to Claire Luu, Rai Manku, Asma Rafiq, Ailar Javadi, Lefke Andreou, Arman Abrahamyan, Netta Wilkie, Peg McCarthy, and Murdoch Wilkie. The Buildings Management crew and Hainaught Services for keepin' it real. The support staff from the School of Geography, Andrew Loveland, Dan Israel, Ayesha Sabri, Elenor Schriber, Emma Schapcott, Helen Mclurg, Lis Parnham, and Anna Dullic- Sills. And the team of The E-Learning Unit, Stella Ekebuisi, Brett Lucas, Alysa Bram- ble, Gil Ritchie, Manoj Singh, Anisa Patel, Richard Chantler, Rume Begum, and Shelly Reader. This research was funded by Queen Mary, The University of London. Contents Abstract1 Acknowledgements3 Table of Contents4 List of Figures8 List of Tables 10 List of Abbreviations and Symbols 12 1 Introduction 13 1.1 Research Questions.............................. 15 1.2 Thesis Structure............................... 15 1.3 Contributions................................. 17 1.4 Publications.................................. 18 2 Background 19 2.1 Acoustics and Psychoacoustics of a Moving Object............ 20 2.1.1 Amplitude Level and the Inverse Square Law........... 20 2.1.2 The Doppler Shift.......................... 22 2.1.3 Surface Reflections.......................... 22 2.1.3.1 Reverberation....................... 23 2.1.3.2 Direct-to-Reverberant Energy Ratio........... 24 2.1.4 Environmental Attenuation..................... 24 2.1.5 Inter-aural Differences........................ 25 2.1.6 Summary of Acoustics and Psychoacoustics............ 26 2.2 Psychological Investigations of Looming.................. 27 2.2.1 Ecological Validity.......................... 27 2.2.2 Unimodal Looming.......................... 28 2.2.2.1 Visual Looming....................... 28 2.2.2.2 Auditory Looming..................... 31 2.2.3 Multimodal Auditory-Visual Looming............... 32 2.2.4 Sound Sources............................ 34 4 2.2.5 Audio Cues.............................. 36 2.2.5.1 Single Cues......................... 36 2.2.5.2 Audio Cue Hierarchy................... 41 2.2.5.3 Multiple Cues....................... 42 2.2.6 Summary of Psychological Research................ 43 2.3 Industry Application............................. 44 2.3.1 Film.................................. 44 2.3.2 Gaming................................ 45 2.3.3 Simulators and Training Systems.................. 46 2.3.4 Vehicle Technology.......................... 46 2.3.4.1 Electric Cars........................ 46 2.3.4.2 Driver Auditory Feedback................. 47 2.3.5 Summary of Industry Application................. 47 2.4 Chapter Summary.............................. 48 Experiments & Analyses 49 3 A Feature Analysis Study of the Audio Cues in Film Looming Scenes 49 3.1 Aim...................................... 50 3.2 Hypotheses.................................. 50 3.3 Method.................................... 50 3.3.1 Scene Selection Criteria....................... 51 3.3.2 Stimuli................................ 52 3.4 Results..................................... 52 3.4.1 Amplitude............................... 52 3.4.1.1 Magnitude of the Amplitude Increase.......... 52 3.4.1.2 Amplitude Envelope Slope................ 55 3.4.1.3 Amplitude Levels...................... 57 3.4.1.4 Object Velocity (According to the Inverse Square Law) 59 3.4.2 Pan Position............................. 62 3.4.3 Spectral Content........................... 64 3.4.3.1 Spectral Centroid..................... 64 3.4.3.2 Spectral Spread...................... 68 3.5 Discussion................................... 70 4 Responses to Designed Film Looming Stimuli 72 4.1 Aim...................................... 73 4.2 Hypotheses.................................. 73 4.3 Method.................................... 73 4.3.1 Design................................. 73 4.3.2 Participants.............................. 74 4.3.3 Stimuli................................ 74 4.3.4 Apparatus............................... 74 5 4.3.5 Dependent Variable Measurement................. 75 4.3.6 Procedure............................... 77 4.4 Results..................................... 78 4.4.1 Presentation............................