ABSTRACT Title of Document: MEG, PSYCHOPHYSICAL AND
Total Page:16
File Type:pdf, Size:1020Kb
ABSTRACT Title of Document: MEG, PSYCHOPHYSICAL AND COMPUTATIONAL STUDIES OF LOUDNESS, TIMBRE, AND AUDIOVISUAL INTEGRATION Julian Jenkins III, Ph.D., 2011 Directed By: Professor David Poeppel, Department of Biology Natural scenes and ecological signals are inherently complex and understanding of their perception and processing is incomplete. For example, a speech signal contains not only information at various frequencies, but is also not static; the signal is concurrently modulated temporally. In addition, an auditory signal may be paired with additional sensory information, as in the case of audiovisual speech. In order to make sense of the signal, a human observer must process the information provided by low-level sensory systems and integrate it across sensory modalities and with cognitive information (e.g., object identification information, phonetic information). The observer must then create functional relationships between the signals encountered to form a coherent percept. The neuronal and cognitive mechanisms underlying this integration can be quantified in several ways: by taking physiological measurements, assessing behavioral output for a given task and modeling signal relationships. While ecological tokens are complex in a way that exceeds our current understanding, progress can be made by utilizing synthetic signals that encompass specific essential features of ecological signals. The experiments presented here cover five aspects of complex signal processing using approximations of ecological signals : (i) auditory integration of complex tones comprised of different frequencies and component power levels; (ii) audiovisual integration approximating that of human speech; (iii) behavioral measurement of signal discrimination; (iv) signal classification via simple computational analyses and (v) neuronal processing of synthesized auditory signals approximating speech tokens. To investigate neuronal processing, magnetoencephalography (MEG) is employed to assess cortical processing non-invasively. Behavioral measures are employed to evaluate observer acuity in signal discrimination and to test the limits of perceptual resolution. Computational methods are used to examine the relationships in perceptual space and physiological processing between synthetic auditory signals, using features of the signals themselves as well as biologically-motivated models of auditory representation. Together, the various methodologies and experimental paradigms advance the understanding of ecological signal analytics concerning the complex interactions in ecological signal structure. MEG, PSYCHOPHYSICAL AND COMPUTATIONAL STUDIES OF LOUDNESS, TIMBRE, AND AUDIOVISUAL INTEGRATION By Julian Jenkins III Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park, in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2011 Advisory Committee: Professor David Poeppel, Chair Professor Catherine E. Carr Associate Professor Jonathan Z. Simon Associate Professor William J. Higgins Associate Professor William J. Idsardi Associate Professor Timothy Horiuchi © Copyright by Julian Jenkins III 2011 Dedication To my mother, Paula Sandra Fullerton Jenkins. And all sentient beings. ii Acknowledgements Gratitude goes to the members of the committee for assistance at various stages of my graduate school career. Thanks also goes out to the various administrative personnel (Lois Reid, Linda Dalo, Cecilia Jordan, KeCia Harper) who assisted with various queries, issues and crises. Lastly, the following people for help in writing, experimental input, analysis, collaboration, other aspects of completing the work presented and various other things (in no order of importance): Maria Chait, Huan Luo, Mary Howard, Ariane Rhone, Phil Monahan, So-One Hwang, Brian Dillon, Mathias Scharinger, Jeff Walker, Max Ehrmann, Pedro Alcocer, Patty Shields, Norbert Hornstein, Art Popper, Sharon Staples, Cognitive Neuroscience of Language Laboratory, C-CEBH and the Departments of Biology and Linguistics iii Table of Contents Dedication ..................................................................................................................... ii Acknowledgements ...................................................................................................... iii Table of Contents ......................................................................................................... iv List of Figures .............................................................................................................. vi Introduction ................................................................................................................... 1 M100 Responses to Two-Frequency Complex Tones .............................................. 3 Elicitation of Audiovisual Steady-State Responses using Pseudo-Speech Signals .. 5 Psychophysical Discrimination and Clustering of Ecologically Approximate Synthetic Signals ....................................................................................................... 7 Preattentive Classification and Physiological Measurement of Ecologically Approximate Synthetic Signals using MEG ............................................................. 9 M100 Responses to Two-Frequency Complex Tones ................................................ 12 Introduction ............................................................................................................. 12 Materials and Methods ............................................................................................ 16 Subjects ............................................................................................................... 16 Threshold Testing................................................................................................ 16 Perceptual Loudness Estimation......................................................................... 16 Stimuli ................................................................................................................. 18 Delivery ............................................................................................................... 19 Recording ............................................................................................................ 20 Data Analysis: Peak RMS and Latency Analysis ................................................ 20 Data Analysis: Dipole Source Estimation .......................................................... 23 Results ..................................................................................................................... 24 Discussion ............................................................................................................... 30 Figures..................................................................................................................... 36 Elicitation of Audiovisual Steady-State Responses using Pseudo-Speech Signals .... 45 Introduction ............................................................................................................. 45 Materials and Methods ............................................................................................ 50 Participants ......................................................................................................... 51 Stimuli ................................................................................................................. 51 Delivery ............................................................................................................... 54 Recording and Filtering ...................................................................................... 55 Sensor Selection from Pre-Test ........................................................................... 55 Onset Response Evaluation ................................................................................. 56 SSR Analysis........................................................................................................ 57 Across-Participant Response Averaging ............................................................ 57 Statistical Analyses ............................................................................................. 58 Participant Head Location ................................................................................. 59 Results ..................................................................................................................... 59 Across-Participant Power Analysis .................................................................... 60 Statistical Summary ............................................................................................ 62 SSR Power Comparisons .................................................................................... 64 iv Discussion ............................................................................................................... 67 Figures..................................................................................................................... 73 Psychophysical Discrimination and Clustering of Ecologically Approximate Synthetic Signals ......................................................................................................... 85 Introduction ............................................................................................................. 85 Part I: Psychophysical Evaluation of Signal Pairs .................................................. 92 Materials