A study on audio-visual interaction: How visual temporal cues influence grouping of auditory events Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Yu Fan Graduate Program in Music The Ohio State University 2019 Dissertation Committee Udo Will, Advisor Arved Ashby Marjorie K.M. Chan 1 Copyrighted by Yu Fan 2019 2 Abstract Music provides us a complex, rapidly changing acoustic spectrum, and when presented with such stimuli, the human auditory system spontaneously groups acoustic elements together. Real-world experience of rhythm is often multisensory, involving both auditory and visual systems, such as when we observe drumming. However, the manner in which auditory grouping behavior is influenced by concurrent temporal cues from visual stimuli is not fully known yet. This study investigates audio-visual interaction in rhythm perception. The central questions addressed are whether visual rhythm influences human’s rhythmic grouping of auditory event sequences, and whether musical training intensifies or weakens this influence. Two experiments were conducted. The preliminary experiment was developed to identify the visual stimuli, natural versus artificial, that would be appropriate to the present inquiry. Participants were recruited for a synchronized tapping test to natural visual stimuli and artificial visual stimuli. The stability of the synchronization and the rate of re- synchronization were measured in a synchronization task. The results show that participants’ synchronization performance of temporal perception was better with natural visual stimuli, which justified the use of a natural visual stimulus as the rhythmic stimulus in the main study on temporal perception. ii The main study testified the audio-visual interaction through a grouping task. Trained musicians and non-musicians were recruited. In the experiment, participants were asked to perform a grouping task through tapping to a cyclical 12-tone drumbeat sequence with different stress patterns that indicated different grouping structure tendencies (either duplets or triplets). While listening, they were asked to simultaneously watch visual stimuli 1) with duple cueing gestures; 2) with triple cueing gestures; or 3) without any cue. The results show that subjects’ duple/triple grouping choice is significantly related to the concurrent visual stimuli with duple/triple cues, and that visual rhythm had a much greater influence on musicians’ auditory grouping behavior than on that of non-musicians. The result of the first experiment, which incorporated novel and realistic visual stimuli in a rhythm perception task, bolsters the rationale of using closer-to-reality experimental designs. The result of the main study demonstrated and highlighted the role of visual sense in multisensory rhythm perception. Moreover, the current study also found that the audio-visual interaction in rhythmic grouping is correlated to long-term music training, and this finding contributes to the body of knowledge on the process of rhythmic perception. iii Dedication This dissertation is dedicated to my family, whose support and love along my PhD journey has been an anchor and inspiration. iv Acknowledgments In my PhD journey, I received a lot of care and help from teachers, classmates, and friends. Here I would like to express my most sincere gratitude to those who have supported me, helped me, and encouraged me throughout this time. First and foremost, I would like to express my deepest gratitude to my supervisor, Dr. Udo Will, a respectable, responsible, and resourceful scholar, who has provided me with valuable guidance in every stage of my research. Without his enlightening instruction and humbling kindness, I could not have completed my thesis. I also extend my thanks to the rest of my committee, Dr. Chan and Dr. Ashby, for their insightful comments and encouragement; to the musicology department at OSU and the School of Music, for providing an inclusive academic atmosphere that allowed me to learn a lot throughout my PhD journey; and to all the experiment participants who joined this study with a cooperative spirit. My sincere appreciation also goes to a four-year scholarship from the Chinese government, which has supported me in completing my PhD program. Last but not least, I’d like to express my thanks to my beloved family for their loving consideration all these years, especially my husband, Zhe Dong, who faithfully supports and encourages me during my study. v Vita 2007–2012......................................................B.A. Musicology, China Conservatory 2012–Present ..................................................Ph.D., The Ohio State University, Columbus Publications Fan, Y., & Will, U. (2016). Musicianship and tone-language experience differentially influence pitch-duration interaction. Proceedings of the 14th ICMPC, San Francisco. Fields of Study Major Field: Music Area of Emphasis: Cognitive Ethnomusicology vi Table of Contents Abstract ............................................................................................................................... ii Dedication .......................................................................................................................... iv Acknowledgments............................................................................................................... v Vita ..................................................................................................................................... vi List of Tables ...................................................................................................................... x List of Figures .................................................................................................................... xi Chapter 1. Introduction ....................................................................................................... 1 1.1. Distinction Between Sensation and Perception ....................................................... 2 1.2. Rhythm Perception is a Multisensory Experience ................................................... 7 1.3. Embodiment Paradigm in Rhythm Perception Study .............................................. 8 1.4. Summary ................................................................................................................ 14 Chapter 2. Clock-Based Timing and Interval Perception ................................................ 15 2.1. Interval as an Element of Auditory Rhythmic Pattern ........................................... 17 2.2. Mechanism and Variability of Interval Perception ................................................ 19 2.3. Interaction Between Perceived Non-Temporal Factors and Interval Perception ... 23 2.4. Visual Interval Perception...................................................................................... 29 2.5. Summary ................................................................................................................ 31 Chapter 3. Entrainment-Based Timing and Pulse Perception ........................................... 33 3.1. Pulse as a Perceptual Phenomena .......................................................................... 33 3.2. Entrainment and Pulse Perception ......................................................................... 35 3.2.1. Entrainment ..................................................................................................... 35 3.2.2. Entrainment-Based Theories and Pulse .......................................................... 39 3.3. Sensorimotor Links in Pulse Perception ................................................................ 43 3.3.1. Sensorimotor Synchronization to Auditory Stimuli ....................................... 45 3.3.2. Sensorimotor Synchronization to Visual Stimuli ........................................... 46 vii Chapter 4. Grouping in Rhythm Perception ..................................................................... 49 4.1. Discussions on the Structure of African Rhythm .................................................. 50 4.2. “Ambiguity” of Grouping in African Rhythm Perception ..................................... 56 4.3. Repetition-Based Approach for Grouping ............................................................. 58 Chapter 5. Audio-Visual Interaction and Experiment Hypothesis ................................... 64 5.1. Previous Studies on Audio-Visual Interaction in Rhythm Perception .................. 65 5.1.1. Alternative Visual Stimuli .............................................................................. 67 5.1.2. Alternative Auditory Rhythmic Pattern .......................................................... 72 5.2. Present Research Questions ................................................................................... 73 5.2.1. Hypothesis of More Efficient Visual Stimulus ............................................... 73 5.2.2. Hypothesis of Visual Influence on Auditory Grouping Perception ................ 75 5.3. Summary ................................................................................................................ 76 Chapter 6. Experiment on Natural/Artificial Visual Rhythm ........................................... 78 6.1 Participants .............................................................................................................. 78 6.2 Material and Stimuli ..............................................................................................