The Effect of Audio Cues and Sound Source Stimuli on Looming
Total Page:16
File Type:pdf, Size:1020Kb
The Effect of Audio Cues and Sound Source Stimuli on Looming Perception Sonia Wilkie Submitted in partial fulfilment of the requirements of the Degree of Doctor of Philosophy School of Electronic Engineering and Computer Science Queen Mary, University of London January 28, 2015 Statement of Originality I, Sonia Wilkie, confirm that the research included within this thesis is my own work or that where it has been carried out in collaboration with, or supported by others, that this is duly acknowledged below and my contribution indicated. Previously published material is also acknowledged below. I attest that I have exercised reasonable care to ensure that the work is original, and does not to the best of my knowledge break any UK law, infringe any third partys copyright or other Intellectual Property Right, or contain any confidential material. I accept that the College has the right to use plagiarism detection software to check the electronic version of the thesis. I confirm that this thesis has not been previously submitted for the award of a degree by this or any other university. The copyright of this thesis rests with the author and no quotation from it or in- formation derived from it may be published without the prior written consent of the author. Signature: Date: Abstract Objects that move in depth (looming) are ubiquitous in the real and virtual worlds. How humans interact and respond to these approaching objects may affect their continued survival in both the real and virtual words, and is dependent on the individual's capacity to accurately interpret depth and movement cues. In computer-generated environments, including hyper and virtual reality, film, and gaming, these cues are often complex sounds with multiple audio cues that are creatively designed for maximum effect. To accurately generate a dynamic and rich perception of looming objects, the design of such complex stimuli should be based on a firm scientific foundation that encompasses what we know about how people visually and aurally perceive events and interac- tions. Conversely, many psychological studies investigating auditory looming depict the ob- ject's movement using simple audio cues, such as an increase in the amplitude, which are applied to tones that are not regularly encountered in the natural world, such as sine, triangle, or square waves. Whilst the results from these studies have provided im- portant information on human perception and responses, technological advances now allow us to present complex audiovisual stimuli and to collect measurements on human perception and responses to real and hyper-real stimuli. The research in this thesis begins to address the gap that exists between the research corpus and industry usage. This is initially accomplished by conducting a feature analysis of the audio cues and complex sounds constructed by sound designers for film scenes presenting objects moving in depth. This is followed by a perceptual study measuring human responses, both physical and emotional, to the complex audio cues designed for the film scenes. Using physical models, we then select a number of audio cues for closer inspection and introduce the parameter of `room reflections’ as an audio cue. We investigate whether or not human responses to various audio cues differ when they are presented individually or in combination, or when they are applied to an artificial (square wave) sound source or a real world sound source. Finally, we test the capacity of these audio cues to bias multimodal auditory-visual perception of an approaching object. Contents Abstract1 Table of Contents3 List of Figures7 List of Tables9 List of Abbreviations and Symbols 11 1 Introduction 12 1.1 Research Questions.............................. 14 1.2 Thesis Structure............................... 14 1.3 Contributions................................. 16 1.4 Publications.................................. 17 2 Background 18 2.1 Acoustics and Psychoacoustics of a Moving Object............ 19 2.1.1 Amplitude Level and the Inverse Square Law........... 19 2.1.2 The Doppler Shift.......................... 21 2.1.3 Surface Reflections.......................... 21 2.1.3.1 Reverberation....................... 22 2.1.3.2 Direct-to-Reverberant Energy Ratio........... 23 2.1.4 Environmental Attenuation..................... 23 2.1.5 Inter-aural Differences........................ 24 2.1.6 Summary of Acoustics and Psychoacoustics............ 25 2.2 Psychological Investigations of Looming.................. 26 2.2.1 Ecological Validity.......................... 26 2.2.2 Unimodal Looming.......................... 27 2.2.2.1 Visual Looming....................... 27 2.2.2.2 Auditory Looming..................... 30 2.2.3 Multimodal Auditory-Visual Looming............... 31 2.2.4 Sound Sources............................ 33 2.2.5 Audio Cues.............................. 35 2.2.5.1 Single Cues......................... 35 3 2.2.5.2 Audio Cue Hierarchy................... 40 2.2.5.3 Multiple Cues....................... 41 2.2.6 Summary of Psychological Research................ 42 2.3 Industry Application............................. 43 2.3.1 Film.................................. 43 2.3.2 Gaming................................ 44 2.3.3 Simulators and Training Systems.................. 45 2.3.4 Vehicle Technology.......................... 45 2.3.4.1 Electric Cars........................ 45 2.3.4.2 Driver Auditory Feedback................. 46 2.3.5 Summary of Industry Application................. 46 2.4 Chapter Summary.............................. 47 Experiments & Analyses 48 3 A Feature Analysis Study of the Audio Cues in Film Looming Scenes 48 3.1 Aim...................................... 49 3.2 Hypotheses.................................. 49 3.3 Method.................................... 49 3.3.1 Scene Selection Criteria....................... 50 3.3.2 Stimuli................................ 51 3.4 Results..................................... 51 3.4.1 Amplitude............................... 51 3.4.1.1 Magnitude of the Amplitude Increase.......... 51 3.4.1.2 Amplitude Envelope Slope................ 54 3.4.1.3 Amplitude Levels...................... 56 3.4.1.4 Object Velocity (According to the Inverse Square Law) 58 3.4.2 Pan Position............................. 61 3.4.3 Spectral Content........................... 63 3.4.3.1 Spectral Centroid..................... 63 3.4.3.2 Spectral Spread...................... 67 3.5 Discussion................................... 69 4 Responses to Designed Film Looming Stimuli 71 4.1 Aim...................................... 72 4.2 Hypotheses.................................. 72 4.3 Method.................................... 72 4.3.1 Design................................. 72 4.3.2 Participants.............................. 73 4.3.3 Stimuli................................ 73 4.3.4 Apparatus............................... 73 4.3.5 Dependent Variable Measurement................. 74 4.3.6 Procedure............................... 76 4 4.4 Results..................................... 77 4.4.1 Presentation............................. 77 4.4.1.1 Presentation × Time-to-Impact.............. 77 4.4.1.2 Presentation × Emotion (Valence / Arousal)...... 79 4.4.1.3 Presentation × Engagement................ 81 4.4.2 Correlations between the Dependent Variables.......... 83 4.4.2.1 Engagement × Time-to-Impact.............. 83 4.4.2.2 Valence / Arousal × Time-to-Impact........... 84 4.4.2.3 Valence / Arousal × Engagement............. 84 4.5 Discussion................................... 85 5 The Effect of Audio Cues and Sound Source Stimuli on the Perception of Approaching Objects 88 5.1 Aims...................................... 89 5.2 Hypotheses.................................. 89 5.3 Method.................................... 90 5.3.1 Design................................. 90 5.3.2 Participants.............................. 91 5.3.3 Stimuli................................ 91 5.3.3.1 Generation of the Audio Cues for Movement Using Slab3D 93 5.3.4 Apparatus............................... 94 5.3.5 Dependent Variable Measurement................. 94 5.3.6 Procedure............................... 95 5.4 Results..................................... 96 5.4.1 Audio Cues.............................. 96 5.4.1.1 Audio Cues × Time-to-Contact.............. 96 5.4.1.2 Audio Cues × Emotion (Valence / Arousal)....... 98 5.4.1.3 Audio Cues × Engagement................ 100 5.4.1.4 Amplitude Level Presentation............... 102 5.4.2 Sound Source............................. 104 5.4.2.1 Sound Source × Time-to-Contact............. 105 5.4.2.2 Sound Source × Emotion (Valence / Arousal)...... 106 5.4.2.3 Sound Source × Engagement............... 108 5.5 Discussion................................... 109 6 Responses to Complex Auditory-Visual Looming 111 6.1 Aims...................................... 112 6.2 Hypotheses.................................. 112 6.3 Method.................................... 113 6.3.1 Design................................. 113 6.3.2 Participants.............................. 114 6.3.3 Stimuli................................ 114 6.3.4 Apparatus............................... 116 5 6.3.5 Dependent Variable Measurement................. 116 6.3.6 Procedure............................... 116 6.4 Results..................................... 117 6.4.1 Audio Cues.............................. 117 6.4.1.1 Audio Cues × Time-to-Contact.............. 118 6.4.1.2 Audio Cues × Emotion (Valence / Arousal)....... 121 6.4.1.3 Audio Cues × Engagement................ 123 6.4.2 Sound Source............................