RECOGNIZING PHONEMES AND THEIR DISTINCTIVE FEATURES IN THE BRAIN A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Rui Wang March, 2011 © 2011 by Wang Rui. All Rights Reserved. Re-distributed by Stanford University under license with the author. This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States License. http://creativecommons.org/licenses/by-nc/3.0/us/ This dissertation is online at: http://purl.stanford.edu/wj289qm5838 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Patrick Suppes, Primary Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Stephen Boyd I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Bernard Widrow Approved for the Stanford University Committee on Graduate Studies. Patricia J. Gumport, Vice Provost Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives. iii Abstract How the human brain processes phonemes has been a subject of interest for linguists and neuroscientists for a long time. Electroencephalography (EEG) offers a promising approach to observe neural activities of phoneme processing in the brain, thanks to its high temporal resolution, low cost and noninvasiveness. The studies on Mismatch Negativity (MMN) effects in EEG activities in the 1990s suggested the existence of a language-specific central phoneme representation in the brain. Recent findings using magnetoencephalograph (MEG) also suggested that the brain encodes the complex acoustic-phonetic information of speech into the representations of phonological features before the lexical information is retrieved. However, very little success has yet been reported in classifying the brain activities associated with phoneme processing. In my work, I proposed a classification framework which incorporates Principal Components Analysis (PCA), cross-validation and support vector machine (SVM) methods. The initial classification rates were not very good. Progress was made by using bootstrap aggregation (Bagging) scheme and introducing phase calculations. To calculate phase, I computed the Discrete Fourier Transform (DFT) of the original time-domain signal and kept the angles of the finite sample of frequencies. The resulting EEG spectral representation contains only the phase and frequency information and ignores the amplitudes. Using this method, the accurate rate of classifying averaged test samples of eight consonants improved from 41% to 51%. Furthermore, the qualitative analysis of the similarities between the EEG representations, derived from the confusion matrices, illustrates the invariance of brain and perceptual representation of phonemes. For brain and perceptual representation of consonants, voicing is the most distinguishable feature among voicing, continuant and place of articulation. And the feature vowel-height is more robust than vowel- backness in both brain and perceptual representation of vowels. By extending and further refining these methods, it is likely significant classification of other phonemes and features can be made. iv Acknowledgements First of all, I would like to express my gratitude to my principle advisor, Professor Patrick Suppes, for directing me to this interesting area and giving me his invaluable support and guidance throughout my study. The enthusiasm he has for research is infectious and encouraging. I want to thank Professor Bernard Widrow and Professor Stephen Boyd for helpful advises on both my research and academic progress and very insightful comments on the draft of this dissertation. I would also like to thank Professor Christopher Potts for serving as the chairman of my oral exam and giving valuable suggestions from the perspective of linguistics. I am very fortunate to pursue my Ph.D. degree in a supportive and inspiring environment at Stanford University. Being able to work closely with a group of outstanding researchers is important to make my Ph.D. pursuit productive and enjoyable. I am especially grateful for the members of Suppes Brain Lab. In particular, I would like to acknowledge Marcos Perreau Guimaraes, who gave helpful advises and tips on SVM-with-Bagging methods of EEG classification and similarity analysis discussed in this dissertation. Dik Kin Wong, Logan Grosenick, Claudio Carvalhaes, Acacio de Barros and Lene Harbott gave lots of thoughtful ideas and asked motivating questions in group discussion. Blair Bohannan and Duc Nguyen helped me on collecting EEG data. Finally, I would like to thank my family and my parents for their love and support. v Table of Contents Chapter 1 Introduction ........................................................................................ 1 1.1 Phonemes and distinctive features ............................................................ 1 1.2 Brain activities in phoneme perception .................................................... 7 1.2.1 Measurements of brain activities ...................................................... 7 1.2.2 Brain activities in phoneme perception ............................................. 8 1.3 Motivation and Contribution .................................................................... 9 1.4 Outline of the thesis ................................................................................ 12 Chapter 2 Relevant EEG Data .......................................................................... 13 2.1 Syllables-I data ....................................................................................... 13 2.2 Syllables-III data .................................................................................... 14 2.3 Isolated-vowels data ............................................................................... 17 Chapter 3 Signal Processing Methods for Classifying EEG Data ................. 18 3.1 EEG pre-processing ................................................................................ 18 3.2 Classifiers based on brain-speech mapping ............................................ 22 3.2.1 Methodology ................................................................................... 22 3.2.1.1 Diagram of the classification model ........................................... 22 3.2.1.2 Speech features ........................................................................... 24 3.2.1.3 Parameters search ....................................................................... 25 3.2.1.4 Significance level: p-value ......................................................... 25 3.2.2 Experimental results ........................................................................ 26 3.3 Support Vector Machine (SVM) classifiers ........................................... 29 3.3.1 Methodology ................................................................................... 29 3.3.1.1 SVM with Bootstrap aggregating ............................................... 29 vi 3.3.1.2 Diagram of the classifier ............................................................ 32 3.3.2 Classification results ....................................................................... 38 3.3.2.1 Linear vs. Nonlinear Kernels...................................................... 38 3.3.2.2 Leave-out-one-subject experiment ............................................. 41 3.3.2.3 Experiment on the number of trials to calculate average ........... 42 3.3.2.4 Experiment on classifying individual EEG trials using data from single channel. .................................................................................................. 43 3.4 Summary ................................................................................................. 44 Chapter 4 Frequency Analysis of EEG Signals ............................................... 46 4.1 EEG signals in frequency domain .......................................................... 46 4.2 EEG spectral features ............................................................................. 48 4.3 Classification results ............................................................................... 51 4.3.1 Compare the EEG features based on DFT ...................................... 51 4.3.2 Frequency selection ......................................................................... 54 Chapter 5 Invariant Similarities between Brain and Perceptual Representations of Phonemes .................................................................................... 58 5.1 Psychological experiments on phoneme perception ............................... 58 5.2 Similarity measurements ........................................................................ 59 5.2.1 Semi-Order and Invariant Partial Order of similarities ................... 59 5.2.2 Partition tree of similarities ............................................................. 61 5.3 Experimental data analysis ..................................................................... 62 5.3.1 Vowels ............................................................................................