New Jersey Institute of Technology Digital Commons @ NJIT
Dissertations Theses and Dissertations
Spring 2009 Predictive decoding of neural data Yaroslav O. Halchenko New Jersey Institute of Technology
Follow this and additional works at: https://digitalcommons.njit.edu/dissertations Part of the Computer Sciences Commons
Recommended Citation Halchenko, Yaroslav O., "Predictive decoding of neural data" (2009). Dissertations. 901. https://digitalcommons.njit.edu/dissertations/901
This Dissertation is brought to you for free and open access by the Theses and Dissertations at Digital Commons @ NJIT. It has been accepted for inclusion in Dissertations by an authorized administrator of Digital Commons @ NJIT. For more information, please contact [email protected]. C p r ht W rn n & tr t n
h p r ht l f th Un t d St t ( tl , Un t d St t C d v rn th n f ph t p r th r r pr d t n f p r ht d t r l.
Und r rt n nd t n p f d n th l , l br r nd r h v r th r z d t f rn h ph t p r th r r pr d t n. On f th p f d nd t n th t th ph t p r r pr d t n n t t b “ d f r n p rp th r th n pr v t t d , h l r h p, r r r h. If , r r t f r, r l t r , ph t p r r pr d t n f r p rp n x f “f r th t r b l bl f r p r ht nfr n nt,
h n t t t n r rv th r ht t r f t pt p n rd r f, n t j d nt, f lf ll nt f th rd r ld nv lv v l t n f p r ht l .
l t : h th r r t n th p r ht h l th r In t t t f hn l r rv th r ht t d tr b t th th r d rt t n
r nt n n t : If d n t h t pr nt th p , th n l t “ fr : f r t p t : l t p n th pr nt d l r n h n t n l br r h r v d f th p r n l nf r t n nd ll n t r fr th ppr v l p nd b r ph l t h f th nd d rt t n n rd r t pr t t th d nt t f I r d t nd f lt . ABSTRACT
PREDICTIVE DECODING OF NEURAL DATA
by Yaroslav 0. Halchenko
In the last five decades the number of techniques available for non-invasive functional imaging has increased dramatically. Researchers today can choose from a variety of imaging modalities that include EEG, MEG, PET, SPECT, MRI, and fMRI. This doctoral dissertation offers a methodology for the reliable analysis of neural data at different levels of investigation. By using statistical learning algorithms the proposed approach allows single-trial analysis of various neural data by decoding them into variables of interest. Unbiased testing of the decoder on new samples of the data provides a generalization assessment of decoding performance reliability. Through consecutive analysis of the constructed decoder's sensitivity it is possible to identify neural signal components relevant to the task of interest. The proposed methodology accounts for covariance and causality structures present in the signal. This feature makes it more powerful than conventional univariate methods which currently dominate the neuroscience field. Chapter 2 describes the generic approach toward the analysis of neural data using statistical learning algorithms. Chapter 3 presents an analysis of results from four neural data modalities: extracellular recordings, EEG, MEG, and fMRI. These examples demonstrate the ability of the approach to reveal neural data components which cannot be uncovered with conventional methods. A further extension of the methodology, Chapter 4 is used to analyze data from multiple neural data modalities: EEG and fMRI. The reliable mapping of data from one modality into the other provides a better understanding of the underlying neural processes. By allowing the spatial-temporal exploration of neural signals under loose mo e i g assum io s i emo es o e ia ias i e a a ysis o eu a a a ue o o e wise ossi e o wa mo e miss eci ica io e o ose me o o ogy as ee o ma i e i o a ee a o e sou ce y o amewo k o s a is ica ea i g ase a a a a ysis is amewo k yM A is esc i e i C a e 5 PREDICTIVE DECODING OF NEURAL DATA
by Yaroslav 0. Halchenko
A Dissertation Submitted to the Faculty of New Jersey Institute of Technology in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Computer Science
Department of Computer Science
May 2009 Co y ig © 9 y Ya os a a c e ko A IG S ESE E APPROVAL PAGE
PREDICTIVE DECODING OF NEURAL DATA
Yaroslav 0. Halchenko
Dr. Jason RI,. Wang, Dissertation Co-Advisor Date Professor of Computer Science, NJIT
Dr. Stephen J. Hanson, Dissertation Co-Advisor Date Professor, 'Psychology, Rutgers-Newark
Dr. Zoi-Heleni Michalopoulou, Committee Member Date Professor of Mathematical Sciences & Electrical and Computer Engineering, NJIT
Dr. David Nassimi, Committee Member Date Associate Professor of Computer Science, NJIT
/ Dr. Chengjun Liu, Committee Member Date Associate Professor of Computer Science, NJIT
Dr. Barak A. Pearlmutter, Committee Member Date Professor of Computer Science, Hamilton Institute, National University of Ireland, Maynooth BIOGRAPHICAL SKETCH
Author: Ya os a a c e ko Degree: oc o o i oso y Date: May 9
Undergraduate and Graduate Education: • oc o o i oso y i Com u e Scie ce ew e sey I s i u e o ec o ogy ewa k USA 9 • Mas e o Scie ce i Com u e Scie ce U i e si y o ew Me ico A uque que M USA • Mas e o Scie ce i ase a O oe ec o ic E gi ee i g i y sya S a e U i e si y i y sya Uk ai e 1999 • ac e o s eg ee i ase a O oe ec o ic E gi ee i g i y sya S a e U i e si y i y sya Uk ai e 199
Major: Com u e Scie ce
Presentations and Publications: J. amsey S a so C a so Y a c e ko A o ack a C G ymou Si o ems o causa i e e ce om M I I su missio A o ack Y a c e ko a S a so eco i g e a ge-sca e s uc u e o ai u c io y c assi yi g me a s a es ac oss i i i ua s syc o ogica Scie ce I ess M a ke Y a c e ko Se e e g a M ug es e yM A Ma ua A ai a e o i e a //www ym a o g/ yM A-Ma ua M a ke Y a c e ko Se e e g S a so a y a S o ma yM A A y o oo o o mu i a ia e a e a a ysis o M I a a eu oi o ma ics 7(1 37-53 Ma 9 M a ke Y a c e ko Se e e g E O i e i I ü J. W iege C S e ma a y S a so a S o ma yM A A u i yi g a oac o e a a ysis o eu oscie i ic a a o ie s i eu oi o ma ics 3 (3 9
iv S. J. Hanson and Y. O. Halchenko. Brain reading using full brain support vector machines for object recognition: There is no face identification area. Neural Computation, 20 (2):486-503, February 2008.
S. J. Hanson, C. Hanson, Y. O. Halchenko, T. Matsuka, and A. Zaimi. Bottom-up and top- down brain functional connectivity underlying comprehension of everyday visual action. Brain Struct Funct, 212(3-4):231-44, 2007.
S. J. Hanson, D. Rebbechi, C. Hanson, and Y. O. Halchenko. Dense mode clustering in brain maps. Magn Reson Imaging, 25(9):1249-62, 2007. Y. O. Halchenko, S. J. Hanson, and B. A. Pearlmutter. Multimodal integration: fMRI, MRI, EEG, and MEG. In L. Landini, V. Positano, and M. F. Santarelli, editors, Advanced Image Processing in Magnetic Resonance Imaging, Signal Processing and Communications, chapter 8, pages 223-265. Dekker, 2005. ISBN 0824725425.
S. J. Hanson, T. Matsuka, C. Hanson, D. Rebbechi, Y. O. Halchenko, A. Zaimi, and B. A. Pearlmutter. Structural equation modeling of neuroimaging data: Exhaustive search and Markov chain Monte Carlo. In Human Brain Mapping, 2004. Y. O. Halchenko, S. J. Hanson, and B. A. Pearlmutter. Fusion of functional brain imaging modalities using 1-norms signal reconstruction. In Proceedings of the Annual Meeting of the Cognitive Neuroscience Society, San Francisco, CA, 2004. Y. O. Halchenko, B. A. Pearlmutter, S. J. Hanson, and A. Zaimi. Fusion of functional brain imaging modalities via linear programming. Biomedizinische Technik (Biomedical Engineering), 48(2):102-104, 2004.
L. I. Timchenko, Y. F. Kutaev, A. A. Gertsiy, Y. O. Halchenko, L. V. Zahoruiko, and T. Mansur. Method for image coordinate definition on extended laser paths. In S. B. Gurevich, Z. T. Nazarchuk, and L. I. Muraysky, editors, Optoelectronic and Hybrid Optical/Digital Systems for Image and Signal Processing, volume 4148:1, pages 19-26. SPIE, 2000. L. I. Timchenko, Y. F. Kutaev, A. A. Gertsiy, L. V. Zahoruiko, Y. O. Halchenko, and T. Mansur. Approach to parallel-hierarchical network learning for real-time image sequence recognition. In J. W. V. Miller, S. S. Solomon, and B. G. Batchelor, editors, Machine Vision Systems for Inspection and Metrology VIII, volume 3836:1, pages 71-81. SPIE, 1999.
L. I. Timchenko, A. A. Gertsiy, L. V. Zahoruiko, Y. F. Kutaev, and Y. O. Halchenko. Pre-processing of extended laser path images. In Industrial Lasers and Inspection: Diagnostic Imaging Technologies and Industrial Applications, Munich, Germany, June 1999. EOS/SPIE International Symposium. [3827-26].
L. I. Tymchenko, J. Scorukova, S. Markov, and Y. O. Halchenko. Image segmentation on the basis of spatial connected features. In Visnyk VSTU, volume 4, pages 39-43. VSTU University Press, Vinnytsya, Ukraine, 1998. In Ukrainian. T. B. Martinyuk, A. V. Kogemiako, and Y. 0. Halchenko. The model of associative processor for numerical data sorting. In Visnyk VSTU, volume 2, pages 19-23. VSTU University Press, Vinnytsya, Ukraine, 1997. In Ukrainian.
L. I. Tymchenko, J. Scorukova, J. Kutaev, S. Markov, T. Martynuk, and Y. 0. Halchenko. Method spatial connected segmentation of images. In The Third All-Ukrainian International Conference Ukrobraz, Kijiv, Ukraine, November 1996. In Ukrainian.
vi ii LIST OF TERMS
Abbreviation ANOVA Analysis of Variance BCI Brain Computer Interface BEM Boundary Elements Method BMA Bayesian Model Averaging BOLD Blood Oxygenation Level Dependent DC Direct Current DECD Distributed ECD ECD Equivalent Current Dipole EEG Electroencephalography EIT Electrical Impedance Tomography EMF Electromotive Force F/MEG EEG and/or MEG EM Expectation Maximization EMSI Electromagnetic Source Imaging EPI Echo Planar Imaging ER Event-related ERF Event-related Magnetic Field (in MEG) ERP Event-related Potential (in EEG) FDA Food and Drug Administration FDM Finite Difference Method FEM Finite Elements Method FFA Fusiform Face Area FG Fusiform Gyrus FLIRT FMRIB's Linear Image Registration Tool FMRI Functional MRI FMRIB Oxford Centre for Functional MRI of the Brain FOSS Free and Open Source Software FSL FMRIB Software Library GLM General Linear Model GPR Gaussian Process Regression HbO Oxy-hemoglobin HbR Deoxy-hemoglobin HbT Hemoglobin (Total Count) HR Hemodynamic Response HRF HR Function ICA Independent Component Analysis IED Interictal Epileptic Discharge LCMV Linearly Constrained Minimum Variance LFP Local Field Potential LO Lateral Occipital LP Linear Programming LTIS Linear Time Invariant System MAP Maximum a Posteriori MANOVA Multivariate Analysis of Variance MCMC Monte-Carlo Markov Chain MEG Magnetoencephalography MFG Middle Frontal Gyrus ML Machine Learning MR Magnetic Resonance MRI MR Imaging NIRS Near-Infrared Spectroscopy NMR Nuclear Magnetic Resonance PCA Principal Component Analysis PCC Posterior Cingulate Cortex PD Proton Density Imaging PDF Probability Density Function PET Positron Emission Tomography PCUN Precuneous PHG Parahippocampal Gyrus PLS Partial Least-Squares PyMVPA Python Multivariate Pattern Analysis Toolbox pSTS Posterior Superior Temporal Sulcus RFE Recursive Feature Elimination ROC Receiver Operating Characteristic ROI Region of Interest SMLR Sparse Multinomial Logistic Regression SNR Signal-to-noise Ratio SOBI Second Order Blind Identification SPECT Single Photon Emission Computed Tomography SPM Statistical Parametric Mapping SQUID Superconducting Quantum Interference Device STS Superior Temporal Sulcus STG Superior Temporal Gyrus SVD Singular Value Decomposition SVM Support Vector Machine SVR Support Vector Machine Regression TOFC Temporal Occipital Fusiform Cortex TR Time of Repetition (in fMRI) VEP Visual Evoked Potential WMN Weighted Minimum Norm Symbol O Zero matrix of appropriate dimensionality C Covariance matrix F BOLD fMRI data matrix (N x U) Fi Spatial filter matrix for the i-th dipole (M x L) G General E/MEG lead matrix I Identity matrix ( K Number of simultaneously active voxels K Matrix of correlation coefficients L Number of orthogonal axes for dipole moment components, L E {1 3} M Number of EEG/MEG sensors, i e spatial resolution of low spatial resolution modality N Number of voxels, i e spatial resolution of high spatial resolution modality (fMRI) Q General dipole sources matrix (LN xT) Q Dipole sources strength matrix (N xT) T Number of time points of high temporal resolution modality (EEG, MEG) U Number of time points of low temporal resolution modality (fMRI) X General E/MEG data matrix; can contain EEG or/and MEG data (M xT) General E/MEG lead function, incorporating information for EEG or/and MEG O Dipole sources orientation matrix (LN xT) Deviation
7- General time-frequency transformation function Variance
Function M Matrix transpose M+ Generalized matrix inverse (pseudo-inverse) null M The u s ace of M, the set of vectors {x Mx = 0} diag M The diagonal matrix with the same diagonal elements as M GLOSSARY OF TERMS
Chunk A g ou o a a sam es w ic a e i e e e om e o e sam es is i o ma io is im o a i e co e o a c oss- a i a io oce u e i e e c u k i s cou co es o o i e e su ec s e e ime a u s etc. Cross-validation A ec ique o assessi g ow e esu s o a s a is ica a a ysis wi ge e a i e o a i e e e a ase I is mai y use i se i gs w e e e goa is e ic io a o e wa s o es ima e ow accu a e y a e ic i e mo e wi e o m i ac ice ( esc i io om e Wikipedia). Dataset A com i a io o a a sam es a e a e a i u es (e.g., sam es a e s a c u ks c a e i s etc.). Label A a ue associa e wi a a a sam e (o mu i e sam es I ca co es o o a ce ai ca ego y e e ime a co i io o i case o a eg essio o em o a a ia e o o e e um e s a e s e e o e e i e e mo e a a ea e as o ea a a e use w e c oss- a i a i g e e o ma ce o e ea e Learner A me o ca a e o e isi g a ma i g om e s ace o a a sam es i o e s ace o a e s I a e s e o g o a o -o e e se ea e is ca e a classifier. I a e s a e o e e a e e is a co es o i g is a ce me ic regression. Machine Learning A ie o Com u e Scie ce w ic aims a co s uc i g me o s suc as ea e s o i eg a e a ai a e k ow e ge e ac e om e e is i g a a Mass-univariate A a a ysis sc eme w e a u i a ia e me o a ie o a o se a es ( a ia es o e o se a e a a ime Multivariate a i g o i o i g a um e o i e e e ma ema ica o s a is ica a ia es ( e i i io om e Merriam-Webster Online Dictionary). I s a is ics i esc i es a co ec io o oce u es w ic i o e o se a io a a a ysis o mo e a o e s a is ica a ia e a a ime Neural Data Modality A e ec io o eu a ac i i y co ec e usi g some a ai a e i s ume a me o (e.g., EEG MEG M I etc.).
xi Sensitivity A score assigned to a particular feature with respect to its impact on the learner's decision. Statistical Learning A field of Computer Science which aims at exploiting statistical properties of the data to construct reliable learners (hence it is related to Machine Learning), and to assess their convergence and generalization performances. Univariate Characterized by or depending on only one random variable (Definition from the Merriam-Webster Online Dictionary). In mathematics univariate refers to an expression, equation, function or polynomial of only one variable. Objects of any of these types but involving more than one variable may be called multivariate. In statistics, this term is used to distinguish a distribution of one variable from a distribution of several variables.
ii ACKNOWLEDGMENT
A dissertation is one of the most important milestones in the life of any Ph.D. student. As such, it forms and evolves together with a student, from raw marble into what one hopes is a fine piece of scientific art. As any artwork it cannot stand alone, uninfluenced by the outside world, thus it looks for the inspiration and advice from it's surrounding, which, in turn, plays an important role in the shaping and molding, aspiring for perfection of the said work.
My lengthy journey toward a Ph.D. degree began at the University of New Mexico under the supervision and guidance of Dr. Barak Pearlmutter. He became the ultimate source of inspiration for me, as he constantly encouraged me to grasp knowledge and to dive into the fascinating field of computer science and brain imaging. Aside from purely academic interests, he was the one who introduced me to the Debian GNU/Linux operating system, which I became the proud developer of, during my tenure as a graduate student.
Fine-grained polishing of my research work was delicately done by Dr. Stephen Hanson. I would like to thank him for the continuous stream of interesting projects to tackle and the supply of coffee beans to power up the gray matter early in the mornings. For his patience, advice and stimulating debates that helped me navigate my way to completion of this thesis work. For furnishing my trips to scientific conferences around the globe and setting up the meetings and collaborations with scientists working on the fascinating research.
Also, my scientific journey could not be accomplished without the often rescuing, and in other cases simply encouraging help from Dr. Jason Wang who was always there to help me find the answers to my questions. I would also like to thank the members of my committee Dr. Chengjun Liu, Dr. Eliza Michalopoulou, and Dr. David Nassimi for sharing with me the load of completing my dissertation, and for their skillful advice. Although the universities were helping me to reach my final destination, I know, that it would be impossible to accomplish the mission without constant support and encouragement from my family: My parents Larisa Petrovna Halchenko and Ruslan Grigoryevich Nikolenko, my dear, loving and always supportive wife Yuliya. My daughters Alisa and Zabava who rushed to this world to join their mom in helping me grind the marble of science by providing additional joy and introducing me to another dimension of happiness. A special thanks to our favorite family pet — a cat who thought she was a mini-lion — Anfyska for annoying me a lot, but always leaving room for more. Unfortunately she passed on last year and is sorely missed.
Any journey is easier when you travel together, hence I want to express my gratitude to Mr. Michael Hanke, once a geographically distant but similar minded member of the Debian GNU/Linux community, now a collaborator and a close friend. I would like to thank him for his endless encouragement, ideas, data, and code sharing, and very fruitful joint research venue. Also I want to express my sincere regards to the other members of our PyMVPA team — Per Sederberg and Emanuele Olivetti — for the joy of sharing ideas, and sparing rare lonely evenings in the interesting and fertile discussions and article compositions. I also thank every member of RUMBA research laboratory at Rutgers-Newark for plentiful and joyful discussions and their help on a variety of occasions. A special "thank you" goes out to Dr. Catherine Hanson for her persistent helpful support.
It would be hard to arrive at the destination and furnish any interesting and meaningful results without having any interesting data for the analysis. Hence I want to thank everyone who shared their collected neural datasets: Dr. James V. Haxby, Dr. Christoph S. Herrmann, Dr. Artur Luczak, Dr. Kenneth D. Harris, and Dr. Ingo Fründ. To conclude, I would like to thank all my friends and colleagues thanks to whom this journey reached its destination and National Science and McDonnel Foundations for providing financial support to empower my progress.
xiv A E O CO E S
Ch pt r
1 I O UC IO 1 1 1 eu oscie ce O e iew 1 Mac i e ea i g Me o s i eu oimagi g 1 1 3 Mu imo a A a ysis 17 1 S a is ica ea i g So wa e a eu oimagi g 19 ESEA C ME O O OGY 1 Mac i e ea i g a a a i g 3 3 Ge e a i a io es i g 7 ea u e Se si i i y A a ysis 5 ea u e Se ec io oce u es 9 3 U IMO A A A YSIS O EU OIMAGI G A A 31 3 1 I o uc io 31 3 E ace u a eco i gs 3 3 3 EEG a MEG 3 3 3 1 I o uc io 3 3 3 EEG 37 3 3 3 MEG 1 3 M I 3 1 I o uc io 3 I e i ica io Mu i e Ca ego ies A a ysis 51 3 3 S eci ici y Assessme ace s ouse e a e 5 3 iscussio MU IMO A A A YSIS O M I A EEG A A 71
xv A E O CO E S (C nt n d
Ch pt r •
1 Mu imo a E e ime ac ices 7 1 1 Measu i g EEG u i g M I C a e ges a A oac es 73 1 E e ime a esig imi a io s 7 o wa Mo e i g o O Sig a 7 1 Co o u io a Mo e o O Sig a 79 eu o ysio ogic Co s ai s 1 3 E is i g Mu imo a A a ysis Me o s 3 1 Co e a i e A a ysis o EEG a MEG wi M I 5 3 ecom osi io ec iques 7 3 3 Equi a e Cu e i o e Mo e s 3 i ea I e se Me o s 9 3 5 eam o mi g 91 3 ayesia I e e ce 93 Sugges e Mu imo a A a ysis Me o 97 1 Mo i a io 97 o mu a io o e A oac 9 3 omises 1 1 Com u a io a E icie cy 1 3 5 a i a io o a Simu a e a a 1 3 5 1 Simu a io 1 5 a a e a a io 1 7 5 3 Sig a eco s uc io 11 5 Se si i i y A a ysis 11 A ica io o ea EEG/ M I a a 11
i TABLE OF CONTENTS (Continued) Chapter Page
1 a a e a a io 11 Sig a eco s uc io 117 3 Se si i i y A a ysis 1 7 Co c usio 1 3 5 YM A MAC I E EA I G AMEWO K O EU OIMAGI G 1 5 5 1 esig 1 5 a ase a i g 13 5 3 Mac i e ea i g A go i ms 131 5 E am e A a yses 133 5 1 oa i g a a ase 13 5 Sim e u - ai A a ysis 135 5 3 Mu i a ia e Sea c ig 13 5 ea u e Se ec io 13 5 5 Co c usio 1 SUMMA Y 1 1 Co c usio 1 Co i u io s eyo e isse a io 1 9 3 u u e Wo k 15 A E I A O WA MO E I G O EEG A MEG SIG A S 15 A E I EEG A MEG I E SE O EM 15 1 Equi a e Cu e i o e Mo e s 15 i ea I e se Me o s is i u e EC 155 3 eam o mi g 1 A E I C E AI S O E MU IMO A A A A A YSIS 1 3
xvii TABLE OF CONTENTS (Continued)
Chapter Page
A E I EE O E SOU CE SO WA E GE MA E O E A A YSIS O EU OIMAGI G A A 173
E E E CES 17
iii IS O A ES
bl
3.1 Minimal Error and Associated Voxels Count 59
5.1 Performance of Various Classifiers with and without Feature Selection. . . . 147 D.1 Free Software Germane to the Analysis of Neuroimaging Data. 174 D.2 Free and Open-source Projects in Python Related to Neuroimaging Data Acquisition and Processing 175
xix LIST OF FIGURES
Figure Page
1 1 Si e iew o e uma ai O y ig emis e e is isua i e 5 1 i e e e e s o uma ai i es iga io [1] 1 3 o -i asi e u c io a ai imagi g equi me om sim e EEG o e e si e M o ow s ows e y ica iew o e use equi me a o om ow isua i es y ica a a 1 1 wo ossi e a oac es owa e a a ysis o eu a a a (o e e am e o M I 13 1 Mac i e ea i g amewo k yM A i s wo k ow a esig yM A co sis s o se e a com o e s (g ay o es suc as M a go i ms o a ase s o age aci i ies Eac com o e co ai s o e o mo e mo u es (w i e o es o i i g a ce ai u c io a i y e g c assi ie s u a so ea u e-wise measu es [e g I- E IE ; 3 ] a ea u e se ec io me o s [ ecu si e ea u e e imi a io E; 35 3 ] y ica y a im eme a io s wi i a mo u e a e accessi e oug a u i o m i e ace a ca e e o e e use i e c a gea y i e a y a go i m usi g a c assi ie ca e use wi a y a ai a e c assi ie im eme a io suc as su o ec o mac i e [S M; 17] o s a se mu i omia ogis ic eg essio [SM ; 37] Some M mo u es o i e ge e ic me a a go i ms a ca e com i e wi e asic im eme a io s o M a go i ms o e am e a Mu i-C ass me a c assi ie o i es su o o mu i-c ass o ems e e i a u e yi g c assi ie is o y ca a e o ea wi i a y o ems A i io a y mos o e com o e s i yM A make use o some u c io a i y o i e y e e a so wa e ackages ( ack o es S M c assi ie s a e i e ace o im eme a io i S ogu o I S M yM A o i es a co e ie w a e o access oug a u i o m i e ace yM A o i es a sim e ye e i e i e ace a is s eci ica y esig e o make use o e e a y e e o e so wa e A a a ysis ui om ese asic e eme s ca e c oss- a i a e y u i g em o mu i e a ase s i s a a e ge e a e oug a ious a a esam i g oce u es [e g oo s a i g 3 ] e ai e i o ma io a ou e esu s o a y a a ysis ca e que ie om a y ui i g ock a ca e isua i e oug a ious o i g u c io s su o e y yM A A e a i e y e esu s ca e ma e ack i o e o igi a a a s ace a o ma o u e ocessi g y s ecia i e oo s e so i a ows e ese a y ica co ec io a e e wee e mo u es as e a ows e e o a i io a com a i e i e aces w ic a oug o e ia y use u a e o ecessa i y use i a s a a ocessi g wo k ow
xx LIST OF FIGURES (Continued)
Figure Page
2.2 Terminology for ML-based analyses of fMRI data. The upper part shows a simple block-design experiment with two experimental conditions ( e and ue and two experimental runs ( ack and w i e Experimental runs are referred to as independent c u ks of data and fMRI volumes recorded in certain experimental conditions are data sam es with the corresponding condition a e s attached to them (for the purpose of visualization the axial slices are taken from the MNI152 template downsampled to 3 mm isotopic resolution). The lower part shows an example ROI analysis of that paradigm. All voxels in the defined ROI are considered as ea u es The three-dimensional data samples are transformed into a two-dimensional sam es ea u e representation, where each row (sample) of the data matrix is associated with a certain label and data chunk. 25
3.1 Confusion matrix of SMLR classifier predictions of stimulus from multiple single units recorded simultaneously. The classifier was trained to discriminate between stimuli of five pure tones and five natural sounds. Elements of the matrix (numeric values and color-mapped visualization) show the number of trials which were correctly (diagonal) or incorrectly (off-diagonal) classified by a SMLR classifier during an 8-fold cross-validation procedure. The results suggest a high similarity in the spiking patterns for stimuli of low-frequency pure tones, which lead the classifier to confuse them more often, whereas responses to natural sound stimuli and high-frequency tones were rarely confused with one another. . . . 35
xxi LIST OF FIGURES (Continued)
Figure Page
3 S a is ics o mu i e si g e u i e ace u a simu a eous eco i gs a co es o i g c assi ie se si i i ies A o s swee oug i e e s imu i a o g e ica a is wi s imu i a e s ese e i e mi e o e o s e u e a s ows asic esc i i e s a is ics o s ike cou s o eac s imu us e eac ime i (o e e a e eac u i (o e ig Suc s a is ics seem o ack s imu us s eci ici y o a y gi e ca ego y a a gi e ime oi o u i e owe a o e e s ows e em o a se si i i y o i e o a e ese a i e u i o eac s imu us I s ows a s imu us s eci ic i o ma io i e es o se ca e co e ima i y em o a y ( ew s eci ic o se s wi ma ima se si i i y ike o song2 s imu us o i a s ow y mo u a e a e o s ikes cou s (see 3k s imu us Associa e agg ega e se si i i ies o a u i s o a s imu i i e owe ig igu e i ica e eac u i s s eci ici y o a y gi e s imu us I o i es e e s eci ici y a a sim e s a is ica measu e suc as a ia ce e.g., u i 19 ac i e i a s imu a io co i io s acco i g o i s ig a ia ce u acco i g o e c assi ie se si i i y i ca ies i e i a y s imu i s eci ic i o ma io o a u a so gs 1-3 3 3 3 E e - e a e o e ia s a c assi ie s se si i i ies o e c assi ica io o co o a i e-a co i io s i EEG a a 3 E e - e a e mag e ic ie s a c assi ie se si i i ies e u e a s ows mag e ic ie s o wo e em a y MEG c a e s O e e se so M O ( ig occi i a a o e ig se so M O 1 (ce a occi i a e owe a s ows c assi ie se si i i ies a A O A -sco es o e o e ime o o se so s o c assi ie s s owe equi a e ge e a i a io e o ma ce o a o ima e y % co ec si g e ia e ic io s 3 3 5 S imu i e em a s om [ 5] 5 3 Se si i i y a a ysis o e ou -ca ego y M I a ase e u e a s ows e OI-wise sco es com u e om SM c assi ie weig s a A O A -sco es ( imi e o e ig es a e ee owes sco i g OIs e owe a s ows e og ams wi c us e s o a e age ca ego y sam es (com u e usi g squa e Euc i ea is a ces o o e s wi o - e o SM -weig s a a ma c i g um e o o e s wi e ig es -sco es i eac OI 55 LIST OF FIGURES (Continued)
Figure Page
3.7 Generalization error for all ten subjects on single scans for both kinds of stimuli held out from a training exemplars. Recursive voxel elimination produced a minimum for half the subjects at zero error, while over all subjects on single scans, across both tasks there is nearly a 98% correct generalization on unseen scans. 61 3.8 Voxel sensitivities for brain areas that are diagnostic for FACE (red and on left of each panel) and for HOUSE (blue and right of each panel). In the top left corner we show FFA (fusiform face area as measured by a localizer task) in the first paired panel (same subject, same slice, same task) which shows subject NB1. Below that is subject OB3 also showing FFA in both slices. On the left bottom panel we show PCC which is diagnostic for HOUSE and FACE which both show sensitivity (NB2). The next three paired images show distinctive areas between diagnostic areas of FACE and HOUSE. The next panel above on the right shows pSTS (posterior superior temporal sulcus) which is diagnostic for FACE against HOUSE (subject OB 1), the next panel below shows another FACE diagnostic areas (MFG BA9, subject OB5), and finally in the last paired panel on the right we show a HOUSE diagnostic area (LO) for subject NB5. 64 3.9 Voxel area cluster relative frequencies intersected over all subjects and both tasks. For FACE diagnosticities (top), six areas were identified at p<0.01: Fusiform Gyrus (FG), Parahippocampal Gyrus (PHG), Middle Frontal Gyrus (MFG), Superior Temporal Sulcus (STS), and Precuneus (PCUN), with relative frequencies based total number of voxels (N=363) over all areas. For HOUSE diagnosticities (bottom) there were five areas, (at p<0.01, N=358) some the same, some different, including FG, Parahippocampal Gyrus (PHG), Posterior Cingulate Cortex (PCC), PCUN, and Lateral Occipital (LO). Note that overlapping FACE and HOUSE areas, include FG, PHG PCUN and PCC. Distinctive areas for FACE included STS, MFG, while for HOUSE distinctive areas included only LO. 67 4.1 EEG MR artifact removal using PCA. EEG taken inside the magnet (top); EEG after PCA-based artifact removal but with ballistocardiographic artifacts present (center); EEG with all artifacts removed (bottom). After artifact removal it can be seen that the subject closed his eyes at time 75.9 s. (Courtesy of M. Negishi and colleagues, Yale University School of Medicine) 75 LIST OF FIGURES (Continued) Figure Page
Geome ica i e e a io o su s ace egu a i a io i e MEG/EEG sou ce s ace (A e ce e a co e is i i e i o sou ce e eme s q1 q qK eac e ese i g a EC wi a i e o ie a io A sou ce is i u io s com ose a ec o q i K- ime sio a s ace (B) e sou ce is i u io q is i i e i o wo com o e s ( a E Sa a ge(G T e e mi e y e se si i i y o MEG se so s a q ° c u G w ic oes o o uce a MEG sig a (C e M I ac i a io s e i e a o e su s ace SfMRI ( e su s ace- egu a i e M I-gui e so u io qSS E M is c oses o SfMRI mi imi i g e is a ce II qS SRII II w e e (a N x N iago a ma i wi = 1/ w e e i- M I o e is ac i e/i ac i e is e o ec io ma i i o e o ogo a com eme o S fMRI (A a e om [ igu e 1] 9 3 Ma i g o eu a a a om EEG i o M I u i g ea i g o eac M I o e a si g e ma e is ai e o a co es o i g ime-wi ow o EEG a a o e ic M I sig a i a o e u i g es i g e ai e ma e e ic s M I sig a Com a iso e wee e ic e (i e a ea ime cou se o M I (i g ee a ows o assess e e o ma ce o e ma e 1 Simu a e E i o me S ace 1 o e s i a si g e a e 3 OIs we e e i e o ca y E ac i i y A ows (i e isua i e e o ie a io o i o es wi i g ay ma e o e s ou o 1 EEG se so s a e oca e o e sca su ace 1 5 5 E o e simu a e EEG a a 1 S ec a c a ac e is ics o simu a e EEG sig a s e u e a ese s sam e u a io s o EEG e owe a s ows co es o i g s ec og ams 1 7 7 Sam es o e simu a e a a om a a a mo a i ies a e c ose oca io w ic ca ies e e - e a e ac i i y EEG sig a is e ese e y a se so S 15 w ic is oca e o e o o e " ea " E e - e a e o s (i u e e ic simu a e sig a w ic is cause so e y y E ac i i y o a sig a (i g ee s ows o a sig a cause y a eu a ac i i y w ic i c u es o E a s o a eous com o e s Acqui e sig a (i ack s ows o a sig a wi a i i e oise w ic s ou mimic e ea a a o ai e em i ica y 1 Mu i e mo a eak eque cies o e sis e y ms ge e a e i iso a e eoco e i i o (A a e om [ 1 igu e 1] 1 9
i LIST OF FIGURES (Continued) Figure Page
9 es o e co e a io coe icie s e wee oisy a e ic e M I a a 111 1 es o e co e a io coe icie s e wee c ea a e ic e M I a a 11 11 O igi a a e ic e M I ime se ies o e ese a i e o e s CC s a s o co e a io coe icie (o ea so s co e a io wi a co es o i g sig i ica ce e e MSE/ MS is e a io e wee oo -mea squa e e o ( MSE a oo -mea squa e o e a ge sig a ( MS_ e ec eco s uc io wou co es o o CC=1 a MSE/ MS_ = 113 1 Se si i i ies o S ai e o e ic M I sig a o a o e i OI 1 Eac o co es o s o a si g e c a e o EEG a e ese s se si i i ies i ime ( o i o a a eque cy ( e ica i s 11 13 Agg ega e se si i i y o S o e ig - eque cies o SO EEG e ec o e w i e e ic i g a o e i OI E o - a s co es o o e s a a e o es ima e ac oss s i s o a a 115 1 es o e co e a io coe icie s e wee ea a e ic e M I a a i a si g e su ec om [ 17] 11 15 G M a a ysis sig i ica ( es o e a 1 1 ac i a io (i e a eac i a io (i ue i es o se o au i o y s imu a io S ices a e o e i a io ogica co e io e ce e si e o e ai is ese e o e ig om i e - emis e ic oi 11 1 O igi a a e ic e M I ime se ies o e ese a i e o e s 119 17 Se si i i ies o S ai e o e ic M I sig a o a o e wi ig au i o y co e Eac o co es o s o a si g e c a e o EEG a e ese s se si i i ies i ime ( o i o a a eque cy ( e ica i s 1 1 Agg ega e se si i i y o S o e ow- eque cies o C EEG e ec o e w i e e ic i g a o e i e ig au i o y co e 1 1 19 Agg ega e se si i i y o e eque cies i e a- a a ge o eque cies 1 Agg ega e se si i i y o e sig a o EEG c a e (see igu e C o e y ica oca io o e se so i es ec o e ai 1
xxv LIST OF FIGURES (Continued)
Figure Page
5 1 Sea c ig a a ysis esu s o CA S s SCISSO S c assi ica io o s e e a ii o 1 5 1 a mm (co es o i g o a o ima e y 1 15 a 75 o e s e s e e es ec i e y e u e a s ows ge e a i a io e o ma s o eac a ius A e o ma s a e es o e a i a i y a 35 (c a ce e e 5 a a e o smoo e o e ec e ue u c io a eso u io e ce e o e es e o mi g s e e (i.e., owes ge e a i a io e o i ig em o a usi o m co e o ig a e a occi i a co e is ma ke y e c oss- ai o eac co o a s ice e as e ci c e a ou e ce e s ows e si e o e es ec i e s e e ( o a ius 1 mm e s e e o y co ai s a si g e o e M I-s ace coo i a es (x, y, z) i mm o e ou s e e ce e s a e 1 mm (R1): ( - 1 - 5 mm (R5): ( - 9 - 1 mm (R10): ( -59 -1 a mm (R20): ( -5 - e owe a s ows e ge e a i a io e o s o s e es ce e e a ou ese ou coo i a es us e oca io o e u i a ia e y es e o mi g o e (L1: -35 - 3 - 3; e occi i o- em o a usi o m co e o a a ii e e o a s s ow e s a a e o o e mea ac oss c oss- a i a io o s 137 5 ea u e se ec io s a i i y ma s o e CA S s SCISSO S c assi ica io e co o ma s s ow e ac io o c oss- a i a io o s i w ic eac a icu a o e is se ec e y a ious ea u e se ec io me o s use y e c assi ie s is e i a e 5 1 5% ig es A O A -sco es (A 5% ig es weig s om ai e S M ( E usi g S M weig s as se ec io c i e io (C i e a ea u e se ec io e o me y e SM c assi ie wi e a y e m 1 ( 1 (E a 1 ( A me o s e ia y i e i y a c us e o o e s i e e usi o m co e ce e e a ou M I -3 - 3 - mm A s a i i y ma s a e es o e a i a i y a 5 ( ou o 1 c oss- a i a io o s 1 5 C 1 es o ses use o simu a io ue is e o e use o mo e i g e e - e a e ac i i y w i e a e o e s (i c u i g e ue o e as we we e a i a i y assig e o o e s o ge e a e simu a e es o se 1 C es o se use o M I a a simu a io 1 C 3 Gai ma i o EEG sig a simu a io 1 5
vi LIST OF FIGURES (Continued)
Figure Page
C e i e a io a 1 - sys em see om (A e a ( a o e e ea A = Ea o e C = ce a g = aso a y gea = a ie a = o a = o a o a O = occi i a (C oca io a ome c a u e o e i e me ia e 1 % e ec o es as s a a i e y e Ame ica E ec oe ce a og a ic Socie y ( om [ 59] 1
C 5 Agg ega e se si i i y o e eque cies i e 1 - a a ge o eque cies 1 7
C Agg ega e se si i i y o e eque cies i e δ - a a ge o eque cies 1 C 7 Agg ega e se si i i y o e eque cies i e θ- a a ge o eque cies 1 9 C Agg ega e se si i i y o e sig a o C 1 EEG c a e (see igu e C o e y ica oca io o e se so i es ec o e ai 17 C 9 Agg ega e se si i i y o e sig a o C EEG c a e (see igu e C o e y ica oca io o e se so i es ec o e ai 171 C 1 Agg ega e se si i i y o e sig a o EEG c a e (see igu e C o e y ica oca io o e se so i es ec o e ai 17
xxvii CHAPTER 1
INTRODUCTION
The average Ph.D. thesis is nothing but the transference of bones from one graveyard to another
— Frank J. Dobie "A Texan in England", 1945
The brain is a complex, massively-parallel computing system composed of billions of neurons that it uses to extract, process, and store information about the physical world.
The brain is capable of carrying out complex behavioral and cognitive tasks that range from walking to writing a Ph.D. dissertation. It does this through complex neural networks that recode the physical data available in the world into neural data, that is, it performs the information processing necessary for biological organisms to function in the physical world. The mechanisms that underlie information processing, the specialization of network function, and the interactions among different networks are issues of considerable interest and debate in the neuroscience community.
Neuroscientists can use existing technology to record the epiphenomena associated with neural activity and to extrapolate from this data the mechanisms underlying cognitive processing. These neural data represent large volumes of discrete signals which are complex, noisy, and non-stationary; a rather gross reflection of neural activity across spatial and temporal extents. For this reason, the use of advanced signal-processing methods, although underutilized in neuroscience, could potentially resolve many debates about brain function.
One debate in neuroscience involves the identification of functional specificity in the brain and the mapping of specific modules to a set of cognitive or perceptual processes. Another debate concerns the nature of the interaction among brain modules that facilitate data flow and data processing while engaged in cognitive tasks.
1 Current signal-processing methods used to analyze neural data are geared primarily toward identification. These methods are used to map functional modules or spatio-temporal components of the brain signal specific to some form of information processing, for example early visual processing. Such methods are often able to detect brain areas engaged in a given mental task, but make limited use of neural data, and consequently are often inefficient. Moreover, it remains an open question whether such methods can be used to claim specificity of a given area for a unique aspect of information processing (e.g., recognition of the human face). Because most of the signal-processing methods that are used in neuroscience studies are univariate, any particular basic unit in the temporal and/or spatial dimension of the signal is considered to be independent of any other. Such basic units correspond to a single channel of acquisition in space (e.g., in fMRI), or to a specific temporal offset from stimulus presentation in the temporal domain (e.g., in EEG). Since each basic unit carries a very noisy and information-laden signal, univariate methods are capable of detecting only large gross level effects of neural activation. This feature of univariate approaches requires certain pre-processing steps that necessarily ignore some information embedded in the signal.
Several limitations exist when using univariate methods to analyze neural data. For example, it is often necessary to average registered neural responses over a large number of trials to boost the signal-to-noise ratio. This averaging obfuscates the variance in signals that are not precisely locked to the onset of the stimulus trials. Another problem is that the independence assumption of univariate approaches essentially ignores potential
covariance between neighboring or distant units. A third problem is that identification approaches do not account for various sources of signal variance, nor do they address the causal structure of information processing in the brain. This feature of univariate methods makes them inherently unable to provide adequate mapping between a stimulus context and a given brain state. These problems make univariate methods very limited in their ability to either provide insight into the specific nature of the encoding in each module, or to assess the simi a i y among stimulus conditions. Finally, it should be noted that conventional methods do not promote model testing, which makes it difficult to assess the reliability of the findings. Multivariate methods do exist that can hypothetically account for all the data.
Unfortunately most of the classical multivariate methods (e g multivariate analysis of variance or MANOVA) require a large number of data samples to reliably assess the underlying structure of the signal. Because neural data has high dimensionality, the number of samples that are available is limited. This fact makes it hard (if not impossible) to use classical multivariate methods. In the past decade though, in the domain of statistical learning theory, new methods have been developed. These methods often do not require large number of samples to achieve reasonable performance. Moreover, they inherently provide adequate decoding capabilities together with promoting model testing. These features of statistical learning methods make it possible to develop new strategies for the analysis of neural data. The goal of this dissertation is the construction and analysis of methods that can be used to decode brain signals and that will further our understanding of the mapping between brain state and the experimental conditions. Learning theory methods will be of particular interest in that they are able to decode brain states, and to discover functional structures of the brain signal at a level of detail univariate approaches are not able to provide. These methods are used to examine i e i ica io s eci ici y causa i y and simi a i y features of neural data from the level of neurons to the level of interacting neural systems. Thus, learning theory methods are well suited to the task of revealing the distributed functional networks underlying perceptual and cognitive processing. This dissertation will also provide a new methodology for conjoint multi-modal data analysis. The proposed method provides reliable mapping of the brain state reflected in one acquired data modality (EEG) into that of another data modality (fMRI). Because data in both modalities are convoluted with modality-specific inclusions (e.g., instrumental noise, modality-specific neuro-physiological processes), the reliable mapping between modalities should allow the recovery of any common structure between them, i.e., the effective neuronal activity. The following sections of this chapter will overview existing neural data modalities and provide further reasoning for the exploration of decoding methods using unimodal and multimodal analysis of brain data.
1 1 Neuroscience Overview
Neuroscience is a field of research devoted to the study of the nervous system. The brain (see Figure 1.1) receives the greatest attention from neuroscientists inasmuch as it constitutes the main functional unit of the nervous system. It controls all parts of the body, dictates patterns of behavior, empowers thoughts, and provides the mechanism to acquire, encode, and store knowledge about the world. Hence, brain function research is relevant for anyone with intellectual interest in human behavior, whether that interest is driven philosophically, experimentally, or clinically. Increasing knowledge about brain function also has a pragmatic aspect. Improving the techniques used to decode brain signals can make it easier to "read minds" or to increase the independence of handicapped individuals through brain-computer interfaces. The brain of a human is composed of a network of neurons and other supportive cells (e.g., glial cells) that number in the tens of billions. Each neuron on average has thousands of connections to other neurons which allows them to form functional networks, each responsible for specific tasks from simple reception and transmission of information to much more complex processes underlying human behavior. Thus, neuroscience can be 5
Figure 1.1 Side view of the human brain. Only right hemisphere is visualized. studied at many different levels (as depicted in Figure 1.2), ranging from the molecular level to that of complex neural systems.
The most basic level of investigation in neuroscience happens at the cellular level, where the fundamental questions focus on the mechanisms underlying the ability of neurons to process signals physiologically and electrochemically. Researchers investigate how signals are processed by the dendrites, somas, and axons, and how neurotransmitters and electrical signals are used to conduct information between neurons. At a higher level, neuroscientists focus on networking of neurons including how these networks are formed. At this level of investigation, the goal is to model how particular networks are used to produce the physiological functions that range from simple reflexes to the complex processing involved in learning and memory. Research that is specifically aimed at understanding the role of complex systems and networks in cognitive behavior is designated as "cognitive neuroscience".
While behavioral experiments allow scientists to uncover high-level aspects of human performance and behavior, they are incapable of uncovering the complexities r 1 Different levels of human brain investigation 1-11. 7 i o e i ai ac i a io A a ysis o ai u c io mus e y o a a e i e om e eu a ac i i y i e i i g ai O e ossi e a oac o u e s a i g ai u c io is e use o simu a io e e is a o goi g o ec " ue ai " [ ] w ic aims o simu a e e uma ai ow o e mo ecu a e e usi g e is i g eu a mo e s a a ai a e ig - e o ma ce com u i g aci i ies us a ey a e succee e o y i simu a i g a si g e a eoco ica co um is s uc u e ca e co si e e e sma es u c io a u i o e
eoco e ¹ Suc a co um is a ou mm a as a iame e o 5 mm a co ai s o y a ou eu o s i uma s a eoco ica co um s a e e y simi a i s uc u e u co ai o y 1 eu o s (a 1 sy a ses es i e is a a ceme mo e i g o e w o e uma ai emai s a am i ious a is a goa a e e i ossi e wou o o i e i ec a swe s a ou e ig e - e e u c io a s uc u e o e ai e a e a i e a oac a ai a e o eu oscie is s i e es e i e e o a io o ai u c io is e a a ysis o e eu a sig a s o e ai U o u a e y a is oi i ime e e is o si g e ec o ogy a a ce e oug o a ow a "s a s o " o e com e e eu a sys em a ei e a mo ecu a o eu o a e e E e i e e we e suc ec o ogy a ai a e a o ma o ecise y esc i i g e i io s o eu o s wi i io s o co ec io s amo g em oes o e is owe e e e o e comi g ese ec o ogica o ems wou o magica y eso e e e a es i eu oscie ce W a is equi e a e ew a a a a ysis a oac es w ic a e a e o e ac e e a i o ma io om uge amou s o a a Mo eo e ese a a ysis me o s wou ee o e cus omi e o e e e o i es iga io ei g o owe (i.e., ce u a e wo k as e ques io s ei g aske a e qui e i e e I is o is easo a esea c e s i e es e i a gi e e e o
¹ eoco e is e a o e ai oug o e es o si e o ig e u c io s suc as co scious oug 8 i es iga io a e e e o e s ecia i e a a ysis me o s o ca u e a ocess eu a ac i i y a em o a a s a ia sca es o i e es esea c o uma ai u c io ese e a a icu a c a e ge i a a
o -i asi e mea s o assessi g e c a ac e is ics o eu o- ysio ogica ocesses i si e e ai was ecessa y I asi e ec iques suc as e ace u a a i ace u a eco i gs w ic a e ee use u i esc i i g eu a u c io i g i a ima s a e o a o ia e o e s u y o uma s I asi e ec iques i uma s a e imi e o ose a ci a y o me ica y equi e su ge y suc as a use o ea e i e sy y a o i g ce ai assum io s a ou e a u e o eu o a sig a s eu o a ac i i y ca e measu e a mo e e as iome ica sig a s ese sig a s ca e o ai e oug se e a i e e y es o o -i asi e ai imagi g mo a i ies suc as e ec oe ce a og a y (EEG mag e oe ce a og a y (MEG uc ea mag e ic
eso a ce ( M imagi g (M I 3 osi o emissio omog a y ( E a ea -i a e s ec osco y ( I S ese o -i asi e eu a a a mo a i ies ca e ca ego i e as ei e assi e o ac i e e u ose o e assi e mo a i ies (EEG a MEG is o egis e am ie e i o me c a ges w ic a e cause y eu o a ocesses i si e e ai Ac i e mo a i ies (suc as M I E a I S c ea e a co o a e e i o me w ic c a ges u e i ue ce o u e yi g eu o a a o e ysio ogica ocesses e e o e ac i e mo a i ies usua y o o egis e eu o a ac i i y i ec y u a e ca u e c a ges cause y e eu o a ac i i y e.g., co sum io o co as age s oo o yge a io o c a ges o oo ow Ca u e ai sig a s y ei e assi e o ac i e mo a i ies a e usua y o -s a io a y sig a s is o e y oise a sig a i e e e ce Mo eo e ey ossess c a ac e is ics s eci ic o e ec ique (mo a i y
om Wo e ( (Augus 3 [w ] o i asi e a e a i g o a ec ique a oes o i o e u c u i g e ski o e e i g a o y ca i y [a i asi e] 3 e e m M I ge e a y su s i u e M so a e u ic cou mo e easi y a o a e m o a imagi g mo a i y wi ou e wo " uc ea " i i 9 w ic was use o acqui e i so i is c ucia o a e a c ea u e s a i g o mo a i y s a u e w e e o mi g a a ce sig a a a ysis e o es mo a i y is EEG I as ee wi e y use i esea c a c i ica s u ies si ce e mi - we ie ce u y A oug ic a Ca o (1 -19 is e ie e o a e ee e i s o eco e s o a eous e ec ica ac i i y o e ai e e m EEG i s a ea e i 19 9 w e a s e ge a syc ia is wo ki g i e a Ge ma y a ou ce o e wo a "i was ossi e o eco e ee e e ec ic cu e s ge e a e o e ai wi ou o e i g e sku a o e ic em g a ica y o o a s i o a e " e i s SQUI - ase MEG e e ime wi a uma su ec was co uc e a MI y Co e [3] a e is success u a ica io o imme ma s SQUI se so s o acqui e a mag e o-ca iog am i 19 9 EEG a MEG a e c ose y e a e ue o e ec o-mag e ic cou i g a e m E/MEG wi e use o e e ge e ica y o ei e EEG MEG o o a oge e A oug EEG a MEG a e e a e e e a e some su e i e e ces amo g em [ ] o E/MEG o i e ig em o a eso u io (mi iseco s u a e a ma o imi a io e oca io o eu o a ac i i y ca e a o i - oi wi co i e ce oca i a io o eu a ac i i y om E/MEG a a is usua y e e e o as e ec omag e ic sou ce imagi g (EMSI) a as ee a c a e gi g a ea o esea c o e as cou e o eca es a is ecause suc ec iques measu e e sig a ou si e o e ea a a e c ea e y e su e -im osi io o e ec omag e ic ie s o uce y ac i i y a e e e o eu o s a eu a e wo ks e e o e i o e o o ai e e a coa se s a ia oca i a io o e ac i a e eu a o u a io s a i - ose i e se o em as o e so e U ike E/MEG, M I mo a i y is i e e y ca a e o o i i g a i i o iew o ai s uc u e a u c io a e e e o asic eu a e wo ks e uc ea Mag e ic eso a ce ( M e ec was i e e e y a simu a eous y isco e e y e i oc a E wa u ce i 19 As a co seque ce o is im o a isco e y
Superconducting Quantum Interference Device 1
igu e 1 3 Non-invasive functional brain imaging equipment: from simple EEG to expensive MR. Top row shows the typical view of the used equipment, and bottom row visualizes typical data. they shared a Nobel Prize in Physics in 1952. In 1970, Raymond Damidian came to the conclusion that the key to MR imaging (M I is the structure and abundance of water in the human body. It was Paul Lauterbur in 1973, however, who implemented the concept of tri-plane gradients used for exciting selective areas of the body. P. Lauterbur along with Peter Mansfield were awarded a Nobel Prize in Physiology and Medicine in 2003 for the invention of MRI, which made a huge impact on medical imaging.
Since their first appearance, M I techniques have been evolving. Today image intensity observed in MR images can be determined by various tissue contrast mechanisms such as proton density, T1 and T2 relaxation rates, diffusive processes of proton spin dephasing, loss of proton phase coherence due to variations of magnetic 11 susce i i i y i e issue A oug M I is ca a e o e ec i g a sie o su e c a ges i e mag e ic ie i e co ica issue cause y eu o a ac i a io [5 ] e i ec a ica io o M I o ca u i g u c io a ac i i y emai s imi e ue o a e y ow sig a - o- oise a io (S o is easo M I is o e a e e a a omica I was owa e e o e 19 ce u y w e C a es oy a C a es S e i g o [7] o i e e i s e i e ce su o i g e co ec io e wee eu o a ac i i y a ce e a oo ow O e u e yea s a e e M I ec ique a ecei e a e io i a a omica s u ies Ogawa e a [ ] i was isco e e a M I ca e ec oo eo yge a io usi g co as is i i g ai ow a amewo k o u c io a ai imagi g usi g M I [9-11] ase o ca u i g oo o yge a io e e - e e e ( O sig a wi ou e use o a y eac i e age s; a ma e u c io a M I ( M I e i s u y o -i asi e u c io a ai imagi g mo a i y a e o o i e ic s a ia i o ma io e y ica s a ia eso u io o O M I i uma esea c is 1-3 mm w ic is su icie o o i e i o ma io a ou eu a ac i i y a e e e o eu a e wo ks O e aw ack is a e emo y amic ( oo e a e es o se ags se e a seco s e i e i i g o eu o s maki g e O M I sig a a coa se e ec io o e eu o a ac i i y es i e is imi a io M I s u ies a e i c ease geome ica y o e e as e yea s a ge y ecause e s a ia eso u io o M I is so ig esea c e s i e es e i e u c io o e uma ai o e ocus o o e o a mos a sma su se o mo a i ies is is ue a y ecause o e ki s o ques io s u e i es iga io a a y o e ea y cos o ea i g o a a y e a a usi g o e ec iques owe e e se ec io o a gi e measu eme a oac ca g ea y imi e ki o esea c y o esis a is o mu a e as we as e ques io s a ca e a esse 12
1.2 Machine Learning Methods in Neuroimaging
Some a a ysis ec iques a e ecome de facto s a a s es i e ei imi a io s o i a o ia e assum io s o a gi e a a y e o i s a ce e ge e a i ea mo e (G M as a a o s a is ica a ame ic ma i g (S M a a ysis i e i e is e e a e a oac use i M I a a a a ysis I S M as ec s o eu a sig a ocessi g we e seg ega e i o wo i e e e s e s identification a integration. e o igi a goa o identification was mo i a e y e i e io o "i e i y u c io a y specialized ai es o ses" [1 ] i.e., o i e i y egio s o e ai (a e e e o eu a e wo ks w e e a s a is ica y sig i ica o io o e eu a sig a cou e we esc i e y k ow e e ime a ma i u a io s (see igu e 1 (a Suc o mu a io o e a ge goa a owe sig i ica sim i ica io i e a a ysis o massi e a ase s suc as M I a a y co si e i g eac acqui e s a ia u i (voxel5 i M I e ms as i e e e o a y o e is assum io omo e e use o mass-univariate a a ysis ec iques owe e is a oac is e y es ic i e i e ms o ossi e i i gs [13 1 ] ecause e a a ysis o eac o e is ca ie ou i iso a io om e es o e o u a io ecause o is ac i im ossi e o a G M a a ysis o accou o a y i e ac io wi i a e wee eu a e wo ks e e o e e ac ua esu s o identification a e imi e o e egio s wi u i o m co as ac oss e e ime a co i io s u ess e i e ac io s we e e ici y mo u a e y e e e ime a esig a we e o i e as i u o e G M ue o suc e ia ce u o a ge g oss-e ec s wi i e sig a e G M is o a e o eso e e u c io a s uc u es a e e e o eu a e wo ks y i ue o i s esig e G M is a es a e o i e i y a eas a ici a i g i e encoding o e e ime a a ia es i o e eu a co e o e acqui e eu a a a owe e ou si e e eu oscie ce commu i y esea c i mac i e ea i g (M a s a is ica ea i g eo y i a icu a as s aw e a se o a a ysis ec iques a
5 Voxel as a o ume e eme e ese i g a a ue o a egu a g i i ee ime sio a s ace 13
Figure 1.4 Two possible approaches toward the analysis of neural data (on the example of fMRI). 14 are typically multivariate, flexible (e.g., classification, regression, clustering), powerful
(e.g., linear and non-linear), and generic, and hence often applicable to various data modalities with minor modality-specific pre-processing [see 15, for a tutorial and further reasoning on application of ML methods to the analysis of fMRI data]. Moreover, large parts of the ML community favor the open-source software development model [16, see also MLOSS6 project website], which leads to an increase in scientific progress due to the superior accessibility of information and reproducibility of scientific results. By virtue of advances in statistical learning techniques, various ML methods, such as support vector machines (SVM) [17], became capable of providing unprecedented generalization performance when tested on the data which were not provided during training. The ability to assess performance on data not seen during training provides the means of assessing the validity of a constructed decoder. Surprisingly, many methods retain a high level of generalization performance even when the dimensionality of the input space greatly exceeds the number of available data samples. Features such as these make statistical learning methods a perfect choice to tackle the problem of reliable decoding of neural data with its rich informational content and its multivariate nature. Advantages of statistical learning methods have recently attracted considerable interest throughout the neuroscience community. Neuroscientists have reported compelling results when they applied such methods toward the analysis of fMRI data'.
The major difference of such approaches from the "standard" SPM is the flow of information. In SPM, the description of the experimental conditions is provided as the input to GLM for a voxel by voxel testing of the desired hypothesis (Figure 1.4(a)). When using statistical learning methods, flow of information is usually reversed: entire (or a selected region of interest, or ROI) neural dataset is provided as the input to the
6http://www.mloss.org 7 1n the literature, authors have referred to the application of machine learning techniques to neural data as decoding [18, 19], information-based analysis [e.g., 20] or multi-voxel pattern analysis [e.g., 21-23]. 15 decoder, w ic is ai e o eco e e e ime a co i io s ou o e o se e eu a a a ( igu e 1 ( o e am e wo suc c assi ie - ase s u ies y ay es a ees [ ] a Kami a i a o g [1 ] we e a e o eco e e o ie a io o isua s imu i om M I a a om uma ima y isua co e O ie a io -se ec i e co um s a e o y a ou 3 -7 pm i wi i mo keys a o a mi ime e sca e i uma s o e easo s s a e ea ie u i a ia e me o s a e i ca a e o o i i g s eci ici y i o ma io o a gi e s a ia oca io a e e e o eu a e wo ks owe e s a is ica ea i g me o s ca e use success u y wi a ge a a se s o e ea su e es o se iases w ic wou o e e ec e y u i a ia e a a ysis Kami a i a o g [1 ] ay es a ees [ ] ese sma sig a iases e ese i g oca is i u e co i g o e es o ses a e su icie o isam igua e s imu us o ie a io s es i e e ac a e M I a a we e eco e a 3 mm s a ia eso u io i.e., a muc mo e coa se eso u io a e sca e o e eu a e wo ks o i e es us mu i a ia e me o s ca e use o accou o o y o e a ia ce i sig a ue o e i e ac io s e wee o e s u o e ac i o ma io a ou u c io a s uc u es om a sig a acqui e a a coa se e e o s a ia eso u io O e c assi ie - ase s u ies a e u e ig ig e e s e g o a decoding a oac o e am e c assi ie - ase a a ysis was i s use o i es iga e eu a e ese a io s o aces a o ec s i e a em o a co e a s owe a e e ese a io s o i e e o ec ca ego ies a e s a ia y is i u e a o e a i g a a ey a e a simi a i y s uc u e a is e a e o s imu us o e ies a sema ic e a io s i s [ 5- 7] is isse a io wo k (Sec io 3 3 a so i c u es wo k [ ] is e ami i g e s eci ici y i o ma io e co e i is i u e a e s e o ai e esu s o i e u e u e s a i g o e is i u e simi a i y a e s o ai e i e a a ysis o e a a om [ 5] S a is ica ea i g me o s a e a ica io s i e a a ysis o o e eu a a a mo a i ies as we o i s a ce ai -com u e i e aces ( CI was a ea y esea c 16 direction that heavily relied on the use of ML methods. In the context of BCI, a trained classifier provides predictions about an intended action by a human based on the acquired neural data (usually from E/MEG modalities). Unlike research designed to uncover the functional structure of the brain, BCI work is concerned primarily with the detection of components of brain signals that a trained participant could easily control to perform a desired action (i.e., moving a cursor around the screen). This dissertation research seeks not only to construct a decoder of neural data, but also to enhance understanding of the functional structure of the brain. Hence, Section 3.3.2 not only presents the ability to reliably predict experimental condition, but also shows how a constructed decoder of E/MEG signals could discover spatio-temporal components of the signal relevant to the research question, which are not detected by conventional univariate identification methods. Besides BCI applications that use statistical learning algorithms in E/MEG, a variety of unsupervised statistical learning methods have recently received a lot of attention. Most of unsupervised methods address the problem of extracting "interesting" components of the signal by the means of the signal decomposition (e.g., principal and independent component analysis (PCA and ICA)). These methods are used to decompose an acquired signal into a combination of basic components, where each component represents a specific neural or physiological process, or simply the instrumental noise. On the cellular level, such methods are usually used to decompose neuronal spike trends into a set of independent units that are believed to correspond to the basic elements of neural networks, i.e., neurons. Since such identification of the neurons is driven by unsupervised methods, no specificity information is attributed to any given neuron in the result. Section 3.2 shows how supervised methods in the context of neural data decoding at the cellular level can provide specificity information for each extracted unit thereby making it possible to derive various conclusions about the neural population, signals of which are captured at the given recording site. 17
Unsupervised methods have also been used in pre-processing of data. The simplest application is the removal of the components which can be attributed to noise with some confidence thereby increasing the efficiency of data filtering. Alternatively such methods can be used for dimensionality reduction when only a few out of a number of components are found to be relevant to the experimental design. However, it is often the case that the components of interest are not the ones that account for most of the variance in the signal (see [13] or "Classification of SVD-mapped Datasets" in PyMVPA manual [29]). To summarize, statistical learning methods have been used extensively in various fields of science and more recently in neuroscience. Nevertheless, only a subset of statistical learning methods are used in the analysis of neural data to discover the functional structure of the brain. Chapter 3 presents various strategies for applying ML methods to the analysis of various neural data modalities at different levels of investigation. It starts by addressing the problem of specificity at the neural level with the analysis of extracellular recordings. It then describes the analysis of E/MEG datasets and the advantages of decoding methods over standard univariate approaches in the identification of spatio-temporal components of the signal relevant to the experimental paradigm. The final section address once more the problem of identification and similarity structure analysis in fMRI data.
1.3 Multimodal Analysis
No single neural data modality has yet become the favored choice for resolving all neuroscience questions about brain function. High temporal resolution of E/MEG modalities is crucial in many event-related (ER) experiments but is absent in BOLD fMRI, a modality that delivers superior spatial resolution but poor temporal resolution. At this point in time a methodology that consolidates the information obtained from different brain imaging modalities and maximizes both spatial and temporal resolution 18 does not exist although such information integration would be extremely valuable in a variety of applications. It might provide a consistent and reliable localization of the functional neuronal units and processes with higher spatial and temporal precision that cannot be achieved using any of the existing modalities alone. It can help to understand the inherent structure of the signal: which parts of the observed signal could be attributed to the neural and physiological processes, and which to the instrumental noise, that is specific to a given neural modality in question. That can further lead to the development of cross-modality filtering methods, which could be used as a preprocessing step in the unimodal data analysis.
The main obstacle in the development of multimodal methods involving fMRI nowadays seems to be the absence of a reasonable universal model for hemodynamics, where the neuronal activation is the primary input factor. The naive models are not general enough to explain the variability of the BOLD signal, whereas complex parametric models that rely heavily on a prior knowledge of nuisance parameters (based on biophysical details), almost never have a reliable and straightforward means of estimation. This fact makes it unlikely to use such comprehensive models as reliable generative models of the BOLD signal at the level of basic neural networks. Due to the difficulties in assessing the ground truth of a combined signal in any realistic experiment — a difficulty further confounded by lack of accurate biophysical models of BOLD signal, any cross-modality fusion approach has to be tackled with caution. Nevertheless, various methodologies can already provide basic test-ground to check the validity of the developed fusion methods: analysis of simplistic experiments, which target perception processes, where there is a small number of isolated focal sources of activity, studies on phantom (artificial) brains with known sources of activation. At last, the employment of the methods which enforce testing of the discovered underlying model on yet unseen data (which is a requisite in most of the 19 supervised ML methods), intrinsically provides unbiased estimation of the performance of the suggested method.
To summarize, now it seems to be the right time for the development of fusion methods which are comprising empirically supported models or are flexible enough to incorporate future elaborated models of the BOLD response. A convincing demonstration of obtaining better explanatory power for the signals and increased accuracy of localization by the means of multimodal integration for a complex protocol would constitute a major success in the field. Chapter 4 addresses the development and validation of a multimodal functional brain imaging technique to gain intrinsic advantages of each used modality. Regression methods, devised in the field of statistical learning, are employed for the conjoint analysis of fMRI and EEG data, where fMRI data is explained in terms of neural activity registered by EEG modality. Such approach provides multiple advantages over existing methods which are presented in the same chapter. Proposed methodology is supported by the results obtained on simulated and empirical neural data.
1.4 Statistical Learning Software and Neuroimaging
Despite the aforementioned advantages and promises of statistical learning theory methods for the analysis of neural data, various factors have delayed their adoption for the analysis of neural information. First and foremost, existing conventional techniques are well-tested and often perfectly suitable for the standard analysis of data from the modality for which they were designed. Most importantly, however, a set of sophisticated software packages has evolved over time. That allows researchers to apply conventional and modality-specific methods without requiring in-depth knowledge about low-level programming languages or underlying numerical methods. In fact, most of these packages come with convenient graphical and command line interfaces. Presence of such interfaces abstracts the peculiarities of the methods and allows researchers to focus on designing experiments and on addressing actual research questions without having to develop specialized analyses for each study. While specialized software packages are useful when dealing with specific properties of a single data modality, they limit the flexibility to transfer newly developed analysis techniques to other fields of neuroscience. This issue is compounded by the closed-source, or restrictive licensing of many software packages. That, in fact, further limits software flexibility and extensibility.
Moreover, only a few software packages exist that are specifically tailored towards straightforward and interactive exploration of neuroscientific data using a broad range of
ML techniques: Matlab® MVPA toolbox for fMRI data9 [30] and 3dsvm plugin for AFNI [31]. At present only independent component analysis (ICA), an unsupervised method, seems to be supported by numerous software packages (see 32, for fMRI, and 33, for
EEG data analysis). Therefore, the application of statistical learning analyses strategies to neuroimaging data usually involves the development of significant amount of custom code. Hence, users are typically required to have in-depth knowledge about both data modality peculiarities and software implementation details.
Growing number of published results on using statistical learning methods in neuroimaging, coupled with the absent software implementation of those analysis pipelines, often makes it difficult, if not impossible, to carry out the same analysis on the datasets at hands to check the validity of the method, or barely to reproduce the reported results.
At the same time, Python has become the open-source scripting language of choice in the research community to prototype and carry out scientific data analyses and to develop complete software solutions quickly. It has attracted attention due to its openness,
Closed source commercial product of Ma Wo ks® 91t is possible to use the low-level functions of this toolbox for other modalities. 21 flexibility, and the availability of a constantly evolving set of tools for the analysis of many types of data. Python's automatic memory management, in conjunction with its powerful libraries for efficient computation (NumPy¹ and SciPy¹¹ ) abstracts users from low-level "software engineering" tasks and allows them to fully concentrate their attention on the development of computational methods.
As an interpreted, high-level scripting language with a simple and consistent syntax, a plethora of available modules, easy ways to interface to low-level libraries written in other languages' and high-level computing environments¹³ , Python is the language of choice for solving many scientific computing problems. Despite the fact that it is possible to perform complex data analyses solely within Python, it once again often requires in-depth knowledge of numerous Python modules, as well as the development of a large amount of code to lay the foundation for one's work. Therefore, it would be of great value to have a framework that helps to abstract from both data modality specifics and the implementation details of a particular analysis method. Ideally, such a framework should help to expose any form of data in an optimal format applicable to a broad range of machine-learning methods, and on the other hand provide a versatile, yet simple, interface to plug in additional algorithms operating on the data. Chapter 5 presents PyMVPA — a Python-based framework for multivariate pattern analysis using supervised and unsupervised ML methods. The chapter provides a
short summary of the principal design concepts, and the basic building blocks of the PyMVPA. This software was used for most of the the analysis results presented in this dissertation, and major parts of the source code for the analysis are readily available under FOSS license as part of PyMVPA distribution, or as supplemental material for [23].
¹°http://numpy.scipy.org ¹¹ http://www.scipy.org ¹2e.g., ctypes, SWIG, SIP, Cython ¹¹³e.g., mlabwrap and RPy C A E
ESEA C ME O O OGY
This works every time, provided you're lucky — U k ow sou -ma e
1 M h n rn n
Mac i e ea i g me o s aim a i eg a i g e a ai a e k ow e ge e ac e om e o i e a a so a ey cou e ia y assess e ou come w e e e ew a a is o i e Sig i ica oos i e a ica i i y o mac i e ea i g owa e a a ysis o ea a a came wi e e e o me o s a is ica ea i g eo y Mo eo e a o e me o s u i i e i is isse a io we e e e o e i e omai o s a is ica ea i g e e e ess "mac i e ea i g" is use oug ou e e as a ge e ic e m o a y me o o ogy w ic a ge s o " ai " a e ia e e ic o o a o e a a Mac i e ea i g (M me o s em oye i is isse a io cou e oug i o wo g ou s su e ise a u su e ise A su e ise ea i g me o equi es a e e a a so i cou e uce e ma i g om e a a o o e a e s is s e is ca e learning. e qua i y o e ea e ma i g is a e wa assesse y a yi g i o ew a a a com a i g ma i g esu s o e ue a e s is s e is ca e generalization testing o sim y testing. U su e ise me o s o o equi e e a e i g o e a a o i e o ea i g Usua y ese me o s a e co s uc e o isco e a i e e s uc u e o e i u a a (e.g., ma imum a ia ce i ea su s aces i CA i e e e com o e s i ICA etc.). e e e is s a mu i u e o M me o s ey i e i ei goa s assum io s a ou e u e yi g a a s uc u e i e e ea u e weig i g a se ec io a ge
3 o imi a io c i e io s o imi a io ou i es a ma y o e as ec s e ia e eco i g o e eu a a a w ic is e o ic o is isse a io oes o aim a e o i g a esc i i g ossi e ea i g me o s i e sea c o e es c oice a e i co ce s i se wi e co s uc io o a ge e ic wo k ow me o s a oo s ca a e o e ac i g i o ma io ou o e eu a a a a i e e e e s o i es iga io o uma ai M - ase a a ysis y ica y co sis s o some asic oce u es a a e i e e e o e c assi ica io a go i m o ecisio ocess a was ac ua y use e.g., e o ca cu a io c oss- a i a io o e ic io e o ma ce a ea u e se ec io Some o e as ec s o ea i g me o s ese e i is c a e a e ese e i e e s ec i e o yM A amewo k w ic is esc i e i g ea e e ai i e C a e 5 Mea w i e igu e 1 ese s ui i g ocks o yM A o ig ig a ious s ages o a a a a ysis
t ndl n
y ica y a a ase o M a a ysis co sis s o ee a s data samples, sample attributes a dataset attributes. W i e e a a sam es co ai e ac ua a e s a s a e use o ai i g o a i a io sam e a i u es o a i io a i o ma io o a e sam e asis (see igu e i s a o emos o ese a e e isc e e a e s (c assi ica io o co i uous a ia es ( eg essio a i e eac a a sam e wi a gi e e e ime a co i io a e e y o i e e asis o e ma i g a is co s uc e y e ea i g me o I is o e ecessa y o e i e g ou s o a a sam es o e am e w e e o mi g c oss- a i a io e i e e e ai i g a a i a io se s mus e i e i ie I e case o M I a a w ic co ai s sig i ica o wa em o a co ami a io ac oss sam es e em o a se a a io o ai i g a a i a io a ase s
igu e 1 Mac i e ea i g amewo k yM A i s wo k ow a esig yM A co sis s o se e a com o e s (g ay o es suc as M a go i ms o a ase s o age aci i ies Eac com o e co ai s o e o mo e mo u es (w i e o es o i i g a ce ai u c io a i y e g c assi ie s u a so ea u e-wise measu es [e g I- E IE ; 3 ] a ea u e se ec io me o s [ ecu si e ea u e e imi a io E; 35 3 ] y ica y a im eme a io s wi i a mo u e a e accessi e oug a u i o m i e ace a ca e e o e e use i e c a gea y i e a y a go i m usi g a c assi ie ca e use wi a y a ai a e c assi ie im eme a io suc as su o ec o mac i e [S M; 17] o s a se mu i omia ogis ic eg essio [SM ; 37] Some M mo u es o i e ge e ic me a a go i ms a ca e com i e wi e asic im eme a io s o M a go i ms o e am e a Mu i-C ass me a c assi ie o i es su o o mu i-c ass o ems e e i a u e yi g c assi ie is o y ca a e o ea wi i a y o ems A i io a y mos o e com o e s i yM A make use o some u c io a i y o i e y e e a so wa e ackages ( ack o es S M c assi ie s a e i e ace o im eme a io in S ogu o LIBSVM. yM A o i es a co e ie w a e o access oug a u i o m i e ace yM A o i es a sim e ye e i e i e ace a is s eci ica y esig e o make use o e e a y e e o e so wa e A a a ysis ui from ese asic e eme s ca e c oss- a i a e y u i g em o mu i e a ase s i s a a e ge e a e oug a ious a a esam i g oce u es [e g oo s a i g 3 ] e ai e i o ma io a ou e esu s o a y a a ysis ca e que ie om a y ui i g ock a ca e isua i e oug a ious o i g u c io s su o e y yM A A e a i e y e esu s ca e ma e ack i o e o igi a a a s ace a o ma o u e ocessi g y s ecia i e oo s e so i a ows e ese a y ica co ec io a e e wee e mo u es as e a ows e e o a i io a com a i e i e aces w ic a oug o e ia y use u a e o ecessa i y use i a s a a ocessi g wo k ow 25
Figure 2.2 Terminology for ML-based analyses of fMRI data. The upper part shows a simple block-design experiment with two experimental conditions ( e and ue and two experimental runs ( ack and w i e Experimental runs are referred to as independent c u ks of data and fMRI volumes recorded in certain experimental conditions are data sam es with the corresponding condition a e s attached to them (for the purpose of visualization the axial slices are taken from the MNI152 template downsampled to 3 mm isotopic resolution). The lower part shows an example ROI analysis of that paradigm. All voxels in the defined ROI are considered as ea u es The three-dimensional data samples are transformed into a two-dimensional sam es ea u e representation, where each row (sample) of the data matrix is associated with a certain label and data chunk. is ma a o y is is y ica y ac ie e y s i i g a e e ime i o se e a u s a a e eco e se a a e y I yM A is y e o i o ma io ca e s eci ie y a s ecia c u ks sam e a i u e w e e eac sam e is associa e wi e ume ica i e i ie o i s a a c u k o u (see igu e I is ossi e o c ea e a y um e o au i ia y sam e a i u es i a simi a as io Mos mac i e ea i g so wa e equi es a a o e e ese e i a sim e wo- ime sio a sam es x ea u es ma i (see igu e o om owe e M I a ase s a e y ica y ou - ime sio a A oug i is ossi e o iew eac o ume as a sim e ec o o o e s oi g so ig o es i o ma io a ou e s a ia o e ies o e o ume sam es is is a a icu a y se ious isa a age gi e a s a ia me ics a es ecia y is a ce i o ma io a e o i e es i ai esea c I a i io some a a ysis a go i ms suc as e mu i a ia e sea c ig [ ] make use o is i o ma io w e ca cu a i g s e es o o e s yM A o ows a i e e a oac Eac a ase is accom a ie y a a s o ma io o ma i g a go i m a ese es a e equi e i o ma io a s o es i as a a ase a i u e ese ma e s a ow o i i ec io a a s o ma io s om e o igi a a a s ace i o e ge e ic - ma i e ese a io a ice e sa I e case o M I o umes e ma e i e es eac ea u e wi i s o igi a coo i a es i e o ume I ca o io a y com u e cus omi a e is a ces (e g Ca esia e wee ea u es y aki g o o e si e a coo i a es i o accou Usi g e ma e i e e e se i ec io om ge e ic ea u e s ace i o o igi a a a s ace makes i easy o isua i e e esu s o e a a ysis o e am e ea u e se si i i y ma s ca e easi y o ec e ack i o a 3- o ume a isua i e i a way a is simi a o a s a is ica a ame ic ma 7
3 G n r l z t n t n
u i g a y ica c assi ie - ase a a ysis a a icu a a ase as o e esam e se e a imes o o ai a u iase ge e a i a io es ima e o a s eci ic c assi ie I e sim es case esam i g is o e y s i i g e a ase so a some a is use o a i a io w i e e emai e is use o ai i g is is o e mu i e imes u i a s a e es ima e is ac ie e o e a icu a sam i g oce u e e aus s a ossi e ways o a i io i g o e a a W i e e o e s i i g o a a ase is e y im o a i mig o e o ious ow o o i gi e e o wa em o a co ami a io ac oss ia s a is o uce y e emo y amic es o se u c io io a i g e s ic se a a io o ai i g a a i a io a ase s wi o uce ias i su seque a a yses o e e e a e c assi ie as access o e a a agai s w ic i wi e a i a e o a ess is o em yM A o i es a um e o esam i g a go i ms e mos ge e ic o e is a -M s i e w e e M ou o N a ase c u ks a e c ose as e a i a io a ase w i e a o e s se e as ai i g a a u i a ossi e com i a io s o M c u ks a e aw is im eme a io ca e use o ea e-o e-ou c oss- a i a io u a so o i es u c io a i y use u o oo s a i g oce u es [3 ] A i io a s i e s a e a ai a e a o uce i s - a -seco - a o o -e e s i s ecause a y s i e may ase i s s i i g o a y gi e sam e a i u e i is ossi e o s i a ase s ase o a a c u ks o y . ., s imu us co i io s Mos a go i ms im eme e i yM A ca e a ame e i e wi a s i e maki g em easy o a y wi i i e e ki s o s i i g o c oss- a i a io oce u es As wi o e u c io s a ai a e i yM A e commo i e ace makes i i ia o a cus om s i e s e a ase esam i g u c io a i y i yM A a so eases o - a ame ic es i g o c assi ica io a ge e a i a io e o ma ce ia a a a a omi a io a oac . ., Mo e Ca o e mu a io es i g [39] y u i g e same a a ysis mu i e imes wi e mu e a ase a e s (i e e e y wi i eac a a c u k i is 28
ossi e o o ai a es ima e o e ase i e o c a ce e o ma ce o a c assi ie o some o e se si i i y measu e is a ows a es ima io o s a is ica sig i ica ce (i e ms o - a ues o esu s ac ie e usi g e o igi a ( o - e mu e a ase
2.4 Feature Sensitivity Analysis
A ima y goa o ai -ma i g esea c is o e e mi e w e e i e ai i o ma io is ocesse a s o e U i a ia e a a ysis oce u es y ica y o i e oca i a io i o ma io ecause eac ea u e is es e i e e e y o a o e s I co as c assi ie - ase a a yses i co o a e i o ma io om e e i e ea u e se i o e o e e mi e w e e o o e c assi ie ca e ac su icie i o ma io o e ic e e e ime a co i io om e eco e ai ac i i y A oug c assi ie s ca use e oi sig a o e w o e ea u e se o make e ic io s i is s i im o a o k ow w ic ea u es co i u e o ose co ec e ic io s Some c assi ie s ea i y o i e i o ma io a ou sensitivities, i.e., ea u e-wise sco es measu i g e im ac o eac ea u e o e ecisio ma e y e c assi ie o e am e a sim e a i icia eu a e wo k o a ogis ic eg essio suc as SMLR, ases i s ecisio s o a weig e sum o e i u s Simi a weig s ca a so e e ac e om a y i ea c assi ie i c u i g S Ms I is ossi e o com a e e weig s o e ea u es o eac o e i i u a a was s a a i e ( -sco e e eac i u ea u e o emo e ossi e a ia ce ac o w ic i a i a e suc a i ec com a iso owe e e e a e a so classifier-independent algorithms o com u e ea u ewise measu es W i e eu a e wo k a S M weig s a e i e e y mu i a ia e a ea u e- wise A O A i.e., e ac io o wi i -c ass a ac oss c ass a ia ces is a u i a ia e measu e as is sim e a ia ce o e o y measu es o eac o e o e a c asses I a i io o a sim e A O A measu e yM A o i es i ea S M GPR, a SMLR weig s as asic ea u e se si i i ies As wi e c assi ie s iscusse i e e ious 9 section, a simple and intuitive interface makes it easy to extend PyMVPA with custom measures (e.g., information entropy). The SciPy package, for example, provides a large variety of measures that can be easily used within the PyMVPA framework. PyMVPA provides some algorithms that can be used on top of the basic featurewise measures to potentially increase their reliability. Multiple feature measures can be easily computed for sub-splits of the training data and combined into a single featurewise measure by averaging, t-scoring or rank-averaging across all splits. This is useful for stabilizing measure estimates, for example, when a dataset contains spatially distributed artifacts. While a GLM is rather insensitive to such artifacts because it looks at each voxel individually [40], classifiers are often able to pick up such a signal if it is related to the classification decision. Moreover, if artifacts are not equally distributed across the entire experiment, then computing measures for separate sub-splits of the dataset can be used to identify and reduce their impact on the final measure. In addition to comparing raw sensitivity values, PyMVPA enables researchers to easily conduct noise perturbation analyses, where one measure of interest, such as cross-validated classifier performance, is computed many times with a certain amount of noise added to each feature in turn. Feature sensitivity is then expressed in terms of the difference between computed measures with and without noise added to a feature [see 26, 41, for equivalence analyses between noise perturbation and simple sensitivities for
S VM].
5 t r S l t n r d r
As mentioned above, featurewise measure maps can be easily computed with a variety of algorithms. However, such maps alone do not provide enough information to determine the features that are necessary or sufficient for performing some classification. However, feature selection algorithms can be used to determine the relevant features based on a 3
ea u ewise measu e As suc ea u e se ec io ca e e o me i a a a- i e o c assi ie - i e as io I a a a- i e se ec io ea u es ca e c ose acco i g o some c i e io suc as a s a is ica y sig i ica A O A sco e o e ea u e gi e a a icu a a ase o a s a is ica y sig i ica -sco e o o e a icu a weig ac oss s i s C assi ie - i e a oac es usua y i o e a seque ce o ai i g a a i a io ac io s o e e mi e e ea u e se w ic is o ima wi es ec o some c assi ica io e o (e.g., a s e i e e ea e-o e-ou eo e ica u e - ou a ue I s ou e o e a o e o m u iase ea u e se ec io usi g e c assi ie - i e a oac e se ec io as o e ca ie ou wi ou o se i g e mai a i a io a ase o e c assi ie Amo g e e is i g a go i ms incremental feature search (I S a recursive feature elimination [ E 35 3 ] a e wi e y use [e.g., 1] a o a ai a e wi i yM A ese me o s i e i w a ey use as a s a i g oi a ow ey e o m ea u e se ec io E s a s wi e u ea u e se a a em s o emo e e eas - im o a ea u es u i a s o i g c i e io is eac e I S o e o e a s a s wi a em y ea u e se a seque ia y a s im o a ea u es u i a s o i g c i e io is eac e e im eme a io s o o a go i ms i yM A a e e y e i e as ey ca e a ame e i e wi a a ai a e asic a me a ea u ewise measu es a e s eci ics o e i e a io ocess a e s o i g c i e ia a e o u y cus omi a e esu s ese e i is isse a io e ie o E o im o e ge e a i a io e o ma ce o S M c assi ie s (Sec io 3 3 I o e a a yses SM c assi ie wi i s au oma ic ea u e se ec io ou o e o m e e e ce was use as e i s e e a e a oac w e e e ime sio a i y o e a a was e a i e y a ge C A E 3
U IMO A A A YSIS O EU OIMAGI G A A
u y e e a e ies a e ies a s a is ics u e s o my ie s o ge e syc o ogy
— A a S ooga skie " e ug i a a i " 1979
3 1 I o uc io
is c a e e o es e a ica io o s a is ica ea i g me o s o e a a ysis o eu a a a a i e e e e s o i es iga io I egi s wi a a a ysis o mu i e e ace u a eco i gs (Sec io 3 w e e su e ise ea i g me o s a e use o co i m a a gi e eco i g si e co ai s eu o s s ecia i e o e gi e ask I s ec io o e se si i i ies o e ai e c assi ie s u e e ic es a a ysis esu s C assi ie se si i i y a a ysis o eac e ac e eu o s em o a sig a e o u io a ows o c a ac e i e s eci ici y o a eu o i es ec o a gi e s imu i co i io em o a y ic sig a s suc as E/MEG w ic e ec eu a ac i i y o e w o e ai a e a a y e e i Sec io s 3 3 a 3 3 3 e e e sugges e eco i g a oac makes i ossi e o u a e s a io- em o a com o e s o e sig a e com o e s o o s ow sig i ica ask- e e e c a ge i a a y e y co e io a u i a ia e me o s Sec io 3 co c u es e c a e wi a e e si e a a ysis o a M I a ase w ic ca ies s a ia y ic i o ma io a ou mec a isms o e ai i o e i isua o ec ocessi g
31 32
Mos o e a a ysis ese e i is c a e was ca ie ou wi i e yM A amewo k e com e e si e e iew o e yM A amewo k is ese e i e C a e 5
3.2 Extracellular Recordings
ecause e e ace u a a ase a wi e a a y e i is sec io is u u is e e e e ime a a acquisi io a ame e s a e esc i e e e o com e e ess a com e e si i i y A ima e e ime s we e ca ie ou i acco a ce wi e a io a I s i u e o ea Gui e o e Ca e a Use o a o a o y A ima s a a o e y u ge s U i e si y S ague- aw ey a s (3 -5 g we e a aes e i e wi u e a e (1 5 g/kg a e wi a cus om aso-o i a es ai A e e a i g a 3 mm squa e wi ow i e sku o e e au i o y co e e u a was emo e a a si ico mic oe ec o e co sis i g o eig ou -si e eco i g s a ks ( eu o e us ec o ogies A A o MI was i se e e eco i g si es we e i e ima y au i o y co e es ima e y s e eo a ic coo i a es ascu a s uc u e [ ] a o o o ic a ia io o eque cy u i g ac oss eco i g s a ks a oca e wi i aye as e e mi e y e ec o e e a i i g a e s i e u e o es (3 7 1 3 k a a i e i e e a u a sou s (e ac e om e C " oices o e Swam " a u esou S u io I aca Y we e use as s imu i Eac s imu us a a u a io o 5 ms o owe y 15 ms o si e ce A s imu i we e a e e a e egi i g a e wi a 5 ms cosi e wi ow e a a acquisi io ook ace i a si g e-wa e sou iso a io c am e (IAC o Y wi sou s ese e ee ie ( /ES 1 ucke - a is A ac ua 33
Individual units ¹ were isolated by a semiautomatic algorithm (KlustaKwik² ) followed by manual clustering (Klusters ³ ). Post-stimulus time histograms (PSTH) of spike counts per each unit for all 1734 stimulus onsets were estimated using a bin size of 3.2 ms. To ensure the accurate estimation of PSTHs, only units with a mean firing rate higher than 2 Hz were selected for further analysis, leaving a total of 105 units for the analysis. Because the segregation of recordings from individual units is performed independent of the associated stimuli, i.e., in an unsupervised fashion (ML terminology), the activity of any particular unit can not be easily attributed to a particular stimulus. As can be seen in the top plots of Figure 3.2, the stimulus-wise descriptive statistics of the units presented make it difficult to pair the activity of a given unit at a particular time to a given stimulus. Furthermore, due to the inter-trial variance in the spike counts, it is even more difficult to reliably determine to what stimulus condition a given trial belongs. To eliminate the problems associated with unsupervised learning, an anlysis based on a reliable decoding of neural data, obtained from this limited neural population, was performed. This new analysis allowed the characterization of all extracted units in terms of their specificity to a given stimulus at a given point in time. The analysis involved a standard 8-fold cross-validation procedure run with an
SMLR [SMLR; 37] classifier, which achieved a mean estimated accuracy of 77.57%
across all 10 types of stimuli. This generalization accuracy was well above chance (10%) for all stimulus categories and provided evidence for a differential signal across the 10 stimuli at the recording site. Misclassifications were found primarily for low-frequency stimuli. Pure tones with 3 kHz and 7 kHz were more often confused with each other than were tones with a larger frequency difference (see Figure 3.1), suggesting a high similarity
¹ The term "unit" here refers to a single entity, which was segregated from the recorded data, and is expected to represent a single neuron. ²http://klustakwik.sourceforge.net ³ http://klusters.sourceforge.net 3 in the spiking patterns for these frequencies. Moreover, the greater discriminability for stimuli at higher frequencies suggests that this neuronal population may be specialized for processing of higher frequency tones. In addition to labelling unseen trials with high accuracy, the trained classifier provided sensitivity estimates for each unit, time bin, and stimulus condition (see bottom plots of Figure 3.2). Stimulus specific information that is contained in the spike times, relative to a given stimulus onset, can be determined in the temporal sensitivity profiles of any particular unit (see unit #42 profiles in lower left plot of Figure 3.2). Alternatively, stimulus specificity could be represented as a slowly modulated pattern of spike counts (see 3 kHz stimuli). The aggregate sensitivity (in this case the sum of absolute sensitivities) across all time-bins provides summary statistics for any unit's sensitivity to a given stimulus (see lower right plot of Figure 3.2), and unlike a simple variance measure, the aggregate sensitivity makes it relatively easy to associate any unit to any set of stimuli. An additional advantage provided by this approach is the ability to identify units whose signals do not display a substantial amount of variance, but nevertheless carry a stimulus specific signal ( . ., unit #28 and 30 kHz stimulus).
3 3 EEG nd MEG
3 3 1 Intr d t n
Conventional analysis approaches of E/MEG often rely on averaging over multiple trials to extract statistically relevant differences between two or more experimental conditions
( . ., averaged ERP [43]). This practice of averaging remains dominant in the investigation of neural processes and relies on an assumption that response does not vary across trials of the same experimental condition. In the last decade, though, statistical decomposition methods ( . ., ICA) have been used more frequently in the analysis of E/MEG data. The popularity of these methods is increasing because they provide a way 35
Figure 3.1 Confusion matrix of SMLR classifier predictions of stimulus from multiple single units recorded simultaneously. The classifier was trained to discriminate between stimuli of five pure tones and five natural sounds. Elements of the matrix (numeric values and color-mapped visualization) show the number of trials which were correctly (diagonal) or incorrectly (off-diagonal) classified by a SMLR classifier during an 8-fold cross-validation procedure. The results suggest a high similarity in the spiking patterns for stimuli of low-frequency pure tones, which lead the classifier to confuse them more often, whereas responses to natural sound stimuli and high-frequency tones were rarely confused with one another. 36
Figure 3.2 Statistics of multiple single unit extracellular simultaneous recordings and corresponding classifier sensitivities. All plots sweep through different stimuli along vertical axis, with stimuli labels presented in the middle of the plots. The upper part shows basic descriptive statistics of spike counts for each stimulus per each time bin (on the left) and per each unit (on the right). Such statistics seem to lack stimulus specificity for any given category at a given time point or unit. The lower part on the left shows the temporal sensitivity profile of a representative unit for each stimulus. It shows that stimulus specific information in the response can be coded primarily temporally (few specific offsets with maximal sensitivity like for song2 stimulus) or in a slowly modulated pattern of spikes counts (see 3kHz stimulus). Associated aggregate sensitivities of all units for all stimuli in the lower right figure indicate each unit's specificity to any given stimulus. It provides better specificity than a simple statistical measure such as variance, e.g., unit 19 active in all stimulation conditions according to its high variance, but according to the classifier sensitivity it carries little, if any, stimuli specific information for natural songs 1-3. 37 to filter out irrelevant variance contributions and to choose only the components which may be relevant to the experimental design. This feature makes it possible to carry out analysis of E/MEG data on a per trial basis [44, 45]. Per trial analysis of E/MEG data offers new insight into the associated underlying processes of the data by addressing the variability of neural activation across trials [46, 47]. The decoding of E/MEG signals with supervised learning methods has been used most extensively in the construction of the brain-computer interfaces. Trained classifiers are often capable of detecting a targeted action type and of predicting a subject's goal for that action. The bulk of this work, therefore, is focused on practical application rather than scientific discovery. However, supervised learning methods can be used to unlock some of the mysteries of the brain. For example, one early study [48] optimized a "spatial" linear discriminator (using a predefined time window during training) which provided localization of the corresponding brain area. Even so, such approaches have not been geared toward automatic detection of both spatial and temporal components of the signal relevant to the experimental design. This next section explores the application of statistical learning methods for the per-trial analysis of E/MEG data. It will be demonstrated that sensitivity analysis of the classifiers reveals spatio-temporal components which are not detected using conventional methods such as ERP.
3 3 EEG
The dataset used for the EEG example consists of a single participant from a previously published study on object recognition [49]. In the experiment, participants indicated, for a sequence of images, whether they considered each particular image a meaningful object or just "object-like." This task was performed under two different speed constraints for two sets of stimuli having different statistical properties. EEG was recorded from 31 38 e ec o es a a sam i g a e o 5 usi g s a a eco i g ec iques e ai s o e eco i g oce u e ca e ou i ü e a [ 9] A e ai e esc i io o e s imu i ca e ou i usc e a [5 co o e images] a i e ma e a [51 i e-a ic u es] ü e a [ 9] e o me a wa e e - ase ime- eque cy a a yses o c a e s om a os e io egio o i e es (i.e., o mu i a ia e me o s we e em oye owe e e e mu i a ia e me o s a e use o i e e ia e e wee ia s wi co o e s imu i ( a i g a oa s ec um o s a ia eque cies a a ig e e o e ai a ia s wi ack a w i e i e-a s imu i ( igu e 3 3A is isc imi a io is o ogo a o e e e ime a ask a equi e a ici a s o is i guis e wee o ec a o -o ec s imu i e a a o is a a ysis we e 7 ms EEG segme s s a i g ms io o e s imu us o se o eac ia O y ia s a asse e semi-au oma ic a i ac e ec io oce u e e o me i e o igi a s u y we e i c u e yie i g 5 ia s ( co o a 3 i e-a Eac ia imese ies was ow sam e o ea i g 1 sam e oi s e ia a e ec o e e eac ia was e i e y i c u i g e EEG sig a o a sam e oi s om a c a e s as a sam e o e c assi ie ( 3 ea u es o a i a y a ea u es o eac sam e we e o ma i e o a e o mea a u i a ia ce ( -sco e As e mai a a ysis a s a a - o c oss- a i a io oce u e was a ie o linear support vector machine [ i CS M; 17] sparse multinomial logistic regression [SM ; 37] a Gaussian process regression wi i ea ke e [ i G ; 5 ] c assi ie s A i io a y e mu i a ia e I- E IE [3 ] ea u e se si i i y measu es a o com a iso a u i a ia e a a ysis o a ia ce (A O A -sco e we e com u e o e same c oss- a i a io a ase s i s A ee c assi ie s e o me wi ig accu acy o e i e e e es a ase s ac ie i g % ( i CS M 91 % (SM a 9 % ( i G co ec 39 si g e ia e ic io s es ec i e y owe e o g ea e i e es a e e ea u es eac c assi ie use o e ic io s Usi g yM A i is e y easy o e ac ea u e se si i i y i o ma io om a use c assi ie s igu e 3 3 s ows e com u e se si i i ies om a c assi ie s a measu es e e is a s iki g simi a i y e wee e s a e o e c assi ie se si i i ies o e o e ime a e co es o i g e e - e a e o e ia (E i e e ce wa e e wee e wo e e ime a co i io s ( igu e 3 3A; e am e s ow o e ec o e ü e a [ 9] o a (mea ac oss a ime- oi s ea o og a y o s o e se si i i ies ( e co um o igu e 3 3C e ea s a ig a ia i i y wi es ec o e s eci ici y amo g e mu i a ia e measu es S M G a SM weig s co g ue y i e i y ee os e io e ec o es as ei g mos i o ma i e (SM weig s o i e e ig es co as o a measu es e I- E IE o og a y is muc ess s eci ic a mo e simi a o e A O A o og a y i i s g o a s a ia s uc u e a o e o e mu i a ia e measu es I s ou e o e owe e a ese o og a ies agg ega e i o ma io o e a ime oi s a e e o e o o o i e i o ma io a ou s eci ic em o a EEG com o e s w ic a e ese e i e co es o i g co um s o igu e 3 3C O a icu a i e es is e com a iso e wee e mu i a ia e se si i i ies a e u i a ia e A O A -sco es om 3 ms o ms o owi g s imu us o se ( ig mos co um o igu e 3 3C emo s a es e se si i i ies o og a ies a 37 ms O y e mu i a ia e me o s (es ecia y SM i CS M a i G e ec e a e e a co i u io o e c assi ica io ask o e sig a i is ime wi ow is a e sig a may e e a e o i ac a ia EEG gamma- a es o ses a ac au e a [53] a e o se e a a ou e same ime a a ici a s iewe com e s imu i Gi e a e ese a a a so seem o s ow a simi a e oke gamma- a es o se [ 9] i is ossi e a mu i a ia e me o s a e se si i e o gamma- a ac i i y i eu a a a 40
Figure 3.3 Event-related potentials and classifiers sensitivities for the classification of color and line-art conditions in EEG data. 1
3 3 3 MEG
e e am e MEG a ase [5 ] was co ec e wi e goa o e e mi i g w e e i is ossi e o e ic e ecog i io o ie y ese e a u a sce es om si g e ia MEG- eco i gs o ai ac i i y usi g s a is ica ea i g me o s O eac ia a ici a s saw a ie y ese e o og a (37 ms e ic i g a a u a sce e o owe imme ia e y y a a e mask (1 ms — 1 ms e s o i e a e wee ese a io a mask e ec i e y co ai s e amou o ocessi g a ca e o e [55] a e e o e imi s e um e o ic u es a a ici a s wi a e ecog i e A e e mask was u e o a ici a s e ic e (usi g a u o ess es o se w e e o o ey wou e a e o ecog i e e ic u e a a us ee ese e Imme ia e y a e is e ic io a i a s we e ese e wi ou a u a sce e o og a s a aske o i ica e w ic o e ou sce es a ee ese e (i.e., a ou -a e a i e o ce -c oice e aye ma c o sam e ask e MEG was eco e wi a 151 c a e C Omega MEG sys em om e w o e ea (sam i g a e 5 a a 1 a a ogue ow ass i e w i e a ici a s e o me is ask e ms i e a o e MEG ime se ies a a a was use o e a a ysis s a e a e o se o e ie y ese e sce e a e e e o e e mask was u e o As i e o igi a s u y o y ia s i w ic e ic io s ma c e ecog i io e o ma ce (i.e., RECOG-RECOG, NRECOG-NRECOG) we e a a y e o e ai s a ou e a io a e o is se ec io e s imu us ese a io i o ma io a e eco i g oce u e see iege e a [5 ] I is a a ysis a a om a si g e a ici a ( a e e P1 i e o igi a u ica io was use e MEG imese ies we e i s ow sam e o a e a ia segme s we e c a e -wise o ma i e y su ac i g ei mea ase i e sig a ( e e mi e om a ms wi ow io o sce e o se O y ime oi s wi i e i s ms a e s imu us o se we e co si e e o u e a a ysis e esu i g a ase co sis e o
151 c a e s wi ime oi s eac (7 ea u es a a o a o 9 sam es ( 33 RECOG ia s a 1 NRECOG ia s e o igi a s u y co ai e a a yses ase o S M c assi ie s w ic e ea e y mea s o e s a io- em o a is i u io o e se si i i ies a e e a a y i se o i es e mos isc imi a i e sig a e au o s a so a esse e issue o i e e i g ea i y u a a ce a ase s 4 Gi e is com e e si e a a ysis e goa o e a a ysis ese e e e was o e e mi e i is a a ysis s a egy cou e e ica e wi i yM A As wi e EEG a a a s a a c oss- a i a io oce u e was a ie ( is ime - o usi g i ea S M a SM c assi ie s A i io a y agai u i a ia e A O A -sco es we e com u e o e same c oss- a i a io a ase s i s e S M c assi ie was co igu e o use i e e e -c ass C- a ues 5 sca e wi es ec o e um e o sam es i eac c ass o a ess e u a a ce um e o sam es w ic was o o e y e o igi a au o s Simi a o iege e a [5 ] a seco c oss- a i a io o a a ce a ase s ( y e o mi g mu i e se ec io s o a a om su se o sam es om e a ge RECOG ca ego y was u o c assi ie s e o me a mos i e ica y o e u u a a ce a ase ac ie i g 9% (SM a 31% ( i CS M co ec si g e ia e ic io s ( 3 % i e o igi a s u y igu e 3 s ows sam e imese ies o e c assi ie se si i i ies a e A O A -sco e o wo os e io c a e s ue o e sig i ica i e e ce i e um e o sam es o eac ca ego y i is im o a o o e a e mea ue osi i e a e
( 6 a amou e o 7 % (SM a 7 % ( i CS M es ec i e y e seco
4U a a ce a ase s a e a omi a ca ego y w ic as co si e a y mo e sam es a a y o e ca ego y a o e ia y ea s o e o em w e a c assi ie e e s o assig e a e o a ca ego y o a sam es o mi imi e o a e ic io e o 5 a ame e C i so -ma gi S M co o s a a e-o e wee wi o e S M ma gi a um e o su o ec o s [see 5 o a e a ua io o is a oac ] 6Mea is equa o e accu acy i a a ce se s a is a 5 % c a ce e o ma ce e e wi u a a ce se si es [see 5 o a iscussio o is oi ] 3
igu e 3 Event-related magnetic fields and classifier sensitivities. The upper part shows magnetic fields for two exemplary MEG channels. On the left sensor MRO22 (right occipital), and on the right sensor MZO01 (central occipital). The lower part shows classifier sensitivities and ANOVA F-scores plotted over time for both sensors. Both classifiers showed equivalent generalization performance of approximately 82% correct single trial predictions.
SVM classifier trained on the balanced dataset achieved a comparable accuracy of 76.07% correct predictions (mean across 100 subsampled datasets), which is a slightly larger drop in accuracy when compared to the 80.8% achieved in the original study [refer to table 3 in 541.
Importantly, these results show that an analysis using statistical classifiers can provide stable results that depend not on a particular implementation, but rather on the methods themselves. Moreover, the example pressented here demonstrates that the integrated framework of PyMVPA can achieve these results with much less effort than was needed in the original study. 44
3.4 FMRI
I e as eca e M I ecame e mos o u a ai imagi g mo a i y amo g esea c e s i e es e i e eu a co e a es associa e wi cog i i e ocessi g ee easo s w y M I is a o e o e o e mo a i ies i c u e e ac a M I o i es ig s a ia eso u io i is o -i asi e a a a acquisi io is e a i e y easy A a ie y o e e ime a esig s a a a ysis a oac es a e ee c ea e o a swe s eci ic eu oscie i ic ques io s usi g M I e o owi g sec io o i es a ie o e iew o e s a a a oac o a a y i g M I a a a co e s u e yi g assum io s omises a s o comi gs A so i c u e i is e sec io is a a a ysis aime a isco e i g ca ego y s eci ic ocessi g o isua a a i e ai e s a a a oac is com a e wi eco i g me o s o ose i is isse a io
3.4.1 Introduction
Analysis Methodologies e e ac o s a a a a ysis me o use o a a y e M I a a (a o some e e E/MEG is s a is ica a ame ic ma i g (S M [1 ] is a oac e i es wo as ec s o eu a sig a ocessi g identification a integration. e u ose o i e i ica io is o "i e i y u c io a y specialized ai es o ses" [1 ] i.e., o e ec egio s o e ai (a e e e o eu a e wo ks w e e a s a is ica y sig i ica o io o e eu a sig a ca e we esc i e y k ow e e ime a ac o s e em asis o i e i ica io seg ega e om i eg a io sig i ica y sim i ies e a a ysis o M I a ase s y co si e i g eac s a ia oca io (voxel i M I e ms o e i e e e o a y o e is ea u e makes S M a y e o mass-univariate a a ysis e goa o S M is o make s a is ica i e e ces a ou s a ia y e e e egio s S M e ies o e co oi use o e ge e a i ea mo e (G M a Gaussia a om ie (G eo y (as we as o e me o s suc as o e o i co ec io e G M 45 part of the analysis is used to estimate the parameters of the model per each spatial location in order to explain observed data. The GLM tests the null-hypothesis that there is no effect of the experiment on that location. It is important to emphasize that the GLM relies on the definition of the forward model of the signal, and is therefore limited to linear modeling of a signal. However, the value of using forward modeling of the BOLD fMRI signal is still an open question (as it will be highlighted in Section 4.2). Despite the requirement of forward model specification, being a univariate method, the GLM is relatively immune to misspecification of the forward BOLD model for a wide range of experimental designs.
Some icks (e g addition of first temporal derivative of the model variables) will improve the identification performance of GLM by accounting for sources of variance that are not explicitly modeled. Unfortunately, being a type of mass-univariate analysis, the GLM does not account for any covariance or causal structure of the signal (besides local region covariance in some SPM implementations). Therefore, actual identification is limited to the regions with uniform activation across experimental conditions, and cannot distinguish the patterns of activity at the neural network level. Such disregard of the covariance structure is often acceptable with judicious experimental design where experimental factors are carefully crafted and well controlled. Precise control of the neural responses, though, can be very difficult in the cognitive tasks, since the effect of many top-down processes (e g attention, memory) are difficult to control. However, top-down processes being a pervasive influence in most cognitive tasks, can not be neglected. A side-effect of the mass-univariate approach is the need to use a statistical correction method to resolve multiple comparison problems that ensue when making inferences over a volume of the brain that involves tens or hundreds of thousands of voxels. Because often the goal of a GLM analysis is to detect relatively extended brain regions, low-pass spatial filtering is a common preprocessing step to boost SNR and 46 hence to decrease the number of false positives in the identification task. Indeed it is a valid preprocessing step under assumptions that neuronal population within regions of interest are uniformly engaged within the experimental conditions of interest. But some brain areas, (e.g., high-level visual object processing regions such as fusiform gyrus) are known to be responsive to a variety of stimuli under the right conditions. For such cases, mass-univariate methods lack power to account for or discriminate between spatial patterns of activity. Furthermore, univariate methods are incapable of exploring any similarity structure among different stimuli belonging to the same category. Standard approaches based on the GLM for identification of potentially selective brain regions are neither cross-validated nor optimized for performance [57-59]. In a typical block design study, the GLM is used to fit selective time series, often accounting for less than 50% of the total variance, despite statistically significant excursions from a predefined baseline (note significance of a statistical test does also imply a high degree of variance accounted for in the fit of the linear model). This fact allows for considerable indeterminacy regarding the identification of the condition, selective voxels, or regions. A common conceptual error that occurs when using this type of method is believing that the strength of the regression coefficients reflect selectivity when they are merely indicating the presence of brain tissue activation conditional on the presence of the stimulus exemplar p(VOXEL 'CONDITION> chance) what many in this field have called selective or even specific (i.e., uniquely selective). Furthermore, because a bio-physiological model of the BOLD fMRI signal does not exist, a stronger BOLD response does not necessarily reflect selective processing at a given location. Despite these limitations, SPM has been used to support various scientific hypotheses about various specialized functional modules within the brain with respect to a given task. However, as noted earlier, reliance on the identification of active brain regions means that the integrative functioning of brain regions can not easily be explored or explained. The authors of SPM, suggest that any aspects of integration would need to 7
e ack e wi "a i e e se o multivariate a oac es a e ami e e e a io s i amo g c a ges i ac i i y i o e ai a ea o e s" Mu i a ia e me o s we e o co si e e o e o owi g easo s "(i ey o o su o i e e ces a ou egio a y s eci ic e ec s (ii ey equi e mo e o se a io s a e ime sio o e es o se a ia e (i e um e o o e s a (iii e e i e co e o ime sio e uc io ey a e ess se si i e o oca e ec s a mass-u i a ia e a oac es"[1 ] I s ea o a oac i g mu i a ia e me o s w ic wou e a ia y o u y ca a e o a essi g e a o eme io e co ce s so ca e effective connectivity mo e i g was o ose o a ess integration as ec E ec i e co ec i i y " e e s e ici y o e i ue ce a o e eu a sys em e e s o e a o e " [1 ] o e o m is y e o a a ysis some egio s o i e es ( OIs a e se ec e a e e i e ac io s amo g em a e e e mi e Co e io a y specialized ai egio s w ic a e i s e ec e wi S M a e ake o e e o es i a co s uc e g a e e ges o w ic a e ei e o i e ase o io assum io s o isco e e ia e aus i e o eu is ic sea c Usua y suc g a mo e i g a so equi es a e imi a y representative time series e ac io oce u e o o i e a si g e ime se ies o eac o e i e g a Suc a s e u e sac i ices e i o ma io a e e e o eu a e wo ks So y esig e e ici se a a io o identification a integration u i g e ocessi g o eu a a a makes i u ike y a i e ac io s a e e e o eu a e wo ks wi e e ec e a e mos a ca e e ec e is e isco e y o coa se e e i e ac io s o "s ecia i e " sys ems Co seque y w a e e i o ma io a ou similarity structure a may e is wi o e a ai a e o summa i e e G M-e o ce se a a io o identification a integration, w i e a o ia e o e a a ysis o a a om e e ime s wi a ge co sis e (ac oss ia s a u i o m g oss-e ec s is o a ica e o a e e ime a esig s o is easo o e me o s o a a ysis a icu a y mu i a ia e ec iques may o i e a 48 good alternative. Offered here is a decoding approach which relies on constructing a reliable mapping between fMRI neural data and experimental variables.
Category Specific Processing There is considerable debate in the neuroscience community concerning how information about visual objects are processed in the brain. Some researchers [57, 59, 60] have proposed that the representation of some visual objects are localized in specific brain tissue (e.g., fusiform face area (FFA) for "faces" processing [60]). However, other work in this field [61] indicates that FFA may code more general stimulus properties (e.g., "expertise"). Other perspectives reject the notion of localized function Haxby et al. [25] and suggest instead that object identity is coded in a distributed way across the brain, somewhat like a relief or topographic map, so that a particular pattern of response is associated with a specific object type or token [25]. More recent work along these lines [26] indicates that the category specific codes may actually be combinatoric, allowing the efficient reuse of voxel information (e.g., neural populations) across different types or tokens.
Claims that areas of the brain are specific to a given stimulus category are often derived by the comparison of levels of activation between two or more conditions. So, in [60], the difference in detectability between faces in a pre-localized FFA is at its maximum around 2% above baseline, while for other objects (except whole humans, headless bodies, or animal heads) the difference is closer to 1%; a factor of 2 in detectability which, indeed, is quite impressive. Proponents of this view maintain that although other areas of the brain might detect the presence of a face stimulus, these other areas are not uniquely face selective as is the FFA. However, supporters of this claim are not distinguishing face detection from face identification. Identification is the ability to label a given stimulus as belonging to a specific category. The degree to which errors occur can be captured in a classic confusion matrix which delineates both the discriminability and similarity of 49 various object tokens [62]. Detectability is related to identification to the extent that it is necessary for identification, but it is not sufficient to make specific category assignment.
Luce [63] has formalized this relation in what is known as the Luce Choice Axiom':
(3.l)
where choice identification P(i v) depends on conditional probabilities w( iIv of the category i given the voxel or feature v, relative to all other potential category choices. This type of calculation can be made with strength or detectability measures such as regression coefficients. When it is done correctly (e.g., Haxby et al. [25]), voxels involved in face identification were found to be distributed over much of inferior temporal lobe.
As described earlier in this thesis, work in visual object recognition that relies on fMRI measures often uses the activity of voxels above some baseline activity to infer selectivity. Although this may seem like a benign choice, perseverates a view that selectivity of cortical tissue can be measured unambiguously with normalized BOLD responses. The problem with these measures are actually two-fold. First, when using the GLM, a detection is calculated that relies on a contrast between a particular stimulus
condition and one or more others. This means that voxel activity is used basically as a
regressor. As such, the GLM is not diagnostic or able to identify category membership that is conditionally dependent on that feature. Worse, the intensity of voxel activation, which covaries with the strength of the coefficient, is confounded with the voxel's location. Moreover, if blood flow to cortical regions actually signaled the amount of "work" or "energetics" involved in processing states, then BOLD intensities that were not at peak levels might indicate more efficient processing than those at peak levels. This possibility means that selectivity measures that are designed towards diagnostic
http://en.wikipedia.org/wiki/Luce' s_choice_axiom 50 identification, rather then towards strength of association, could downplay the importance of voxels that had lower intensity but which were actually more reliable predictors. This poses a problem for cognitive neuroscientists who are interested in finding the brain mechanisms that underlie correct classification or identification of a stimulus and not simply in detecting where the greatest neural response occurs.
The behavioral response of a subject who is asked to identify a masked stimulus [64] will produce a standard psychophysical function that reflects changes in the ability of the subject to correctly identify the stimulus as the mask is changed. However, for the associated fMRI data, a GLM analysis of voxel values can identify only those voxels that are associated or most similar to systematic variations of the independent variable ( . ., the mask). Thus, the GLM does not provide the type of voxel identification function most neuroscientists are seeking. This is where statistical learning theory, and statistical classifiers in particular, can be of considerable benefit.
Classifier-based Analysis There has been enormous progress over the last few years in the development of statistical classifiers. Recent work often focuses on problems having extremely large feature spaces (> 100k to lM) and poor signal/noise ratio, often with remarkable out-of-sample generalization [65]. That why these methods may be particularly useful in analyzing fMRI data. The BOLD signal is known to have notoriously low signal/noise gain and to reflect a mixture of various neural signals and
field potentials. There is also growing evidence that the underlying distribution is non-Gaussian [66-68], which means that parametric classifiers may underestimate valid
signal excursions and make approaches . ., Haxby et al. [25] biased toward conservative generalization estimates. In fact, when Hanson et al. [26] compared Haxby's nearest neighbor classifier to neural network classifiers, a conservative cross-validation estimate showed improvements in generalization by as much as 30-40%. Much of the improvement obtained with statistical classification can be attributed to a 51 more appropriate classification function (e.g., specific nonlinearity) as well as feature weighting and/or selection [69].
In order to discover features (areas of the brain) that are responsive to specific stimulus types it is necessary to focus on whole brain classification. To date of [28] researchers in the visual object recognition domain have used SVM (see [70]) in more generic "brain reading" contexts and, with the exception of [31] who used single scans to filter support vectors, no one has attempted whole brain, single scan classification, focusing instead on regions of interest such as the inferior temporal (IT) lobe. On the other hand, a number of researchers in the object recognition field have found evidence that "object-selective" or "face-selective" areas of the brain exist outside of IT [71-73]. However, it still remains unclear whether these "selective" areas uniquely identify certain object types. Do these other brain areas, that are neither FFA or parahippocampal place area (PPA), carry discriminative information about exemplars like faces or houses? At level of category structure do discriminative areas function? As the first step toward answering these questions, Section 3.4.1 describes how statistical learning methods can be used to identify areas engaged in object processing, and to associate a level of semantic processing to those areas (e.g., like that used to distinguish between animate and inanimate categories). Moreover, the existing claim that some brain areas, such as fusiform face area, are category specific is challenged In Section 3.4.3 statistical learning methods are used to demonstrate that category-specific areas do carry discriminative information about nonpreferred stimuli.
3.4.2 Identification: Multiple Categories Analysis
Data from a single subject who participated in a study published by Haxby et al. [25], which has been reanalyzed by a number of researchers since the original publication [13, 26, 28], serves as the example fMRI dataset for the unimodal analysis of fMRI presented 52
Figure 3.5 Stimuli exemplars from [25]. 53 i is isse a io e a ase i se co sis s o 1 u s I eac u e a ici a assi e y iewe g eysca e images (see igu e 3 5 o eig o ec ca ego ies g ou e i s ocks a se a a e y es e io s Eac image was s ow o 5 ms a was o owe y a 15 ms i e -s imu us i e a W o e ai M I a a we e eco e wi a o ume e e i io ime o 5 ms so a ai ac i i y associa e wi a gi e s imu us ock was ca u e y oug y 9 o umes o a com e e esc i io o e e e ime a esig a M I acquisi io a ame e s see a y e a [ 5] i s aw M I a a we e mo io co ec e usi g I om S [7 ] A su seque a a ocessi g was o e wi yM A (see Sec io 5 A e mo io co ec io i ea e e i g was e o me o eac u i i i ua y o a i io a s a ia o em o a i e i g was a ie o e sake o sim ici y e a ase was e uce o a ou -c ass o em ( aces ouses ca s a s oes A o umes eco e u i g a y o ese ocks we e e ac e a o e -wise -sco e is o ma i a io was e o me i i i ua y o eac u o e e a y i o ma io a s e ac oss u s A e e ocessi g e same se si i i y a a ysis use o a o e a a mo a i ies ( ese e i e ious sec io s o C a e 3 was a ie o is a ase e e o y a SM c assi ie was use ( - o c oss- a i a io wi wo o e we e e e ime a u s g ou e i o o e c u k a ai e o si g e M I o umes a co e e e w o e ai o com a iso a u i a ia e A O A was com u e agai o e same c oss- a i a io a ase s i s e SM c assi ie e o me e y we o e i e e e es a ase s co ec y e ic i g e ca ego y o 9 7% o a si g e o ume sam es i e es a ase s o e ami e w a i o ma io was use y e c assi ie o eac is e o ma ce e e egio o i e es ( OI ase se si i i y sco es we e com u e o o -o e a i g s uc u es e i e y e o a i is ic a a -O o co ica a as [75] as s i e wi S [7 ] o c ea e e OIs e o a i i y ma s o a s uc u es we e es o e a 5% a am iguous o e s we e assig e o e s uc u e wi e 54 higher probability. The resulting map was projected into the space of the functional dataset using an affine transformation and nearest neighbor interpolation.
In order to determine the contribution of each ROI, the sensitivity vector was first normalized (across all ROIs), so that all absolute sensitivities summed up to 1
(L 1 -normed). Afterwards, ROI-wise scores were computed by taking the sum of all sensitivities in a particular ROI. The upper part of figure 3.6 shows these scores for the 20 highest-scoring and the three lowest-scoring ROIs. The lower part of the figure shows dendrograms from a hierarchical cluster analysis on relevant voxels from a block-averaged variant of the dataset (but otherwise identical to the classifier training data). For SMLR, only voxels with a non-zero sensitivity were considered in each particular ROI. For ANOVA, only the voxels with the highest F-scores (limited to the same number as for the SMLR case) were considered. For visualization purposes the dendrograms show the distances and clusters computed from the average samples of each condition in each dataset chunk (i.e, two experimental blocks), yielding 6 samples per condition. The four chosen ROIs clearly show four different cluster patterns. The 92 selected voxels in temporal occipital fusiform cortex (TOFC) show a clear clustering of the experimental categories, with relatively large sample distances between categories.
The sensitivity plot (upper part of Figure 3.6) shows that univariate statistics undervalued
the voxels within TOFC and LOC inf, although those are the areas which carry category-specific information. The pattern of the 36 voxels in angular gyrus reveals an animate/inanimate clustering, although with much smaller distances. It is an interesting result given that clustering was done in an unsupervised fashion. However, it is a good demonstration of how a multivariate technique can reveal the presence of information at this level of semantic specificity. The largest group of 148 voxels in the frontal pole ROI seems to have no obvious structure in their samples. Despite that, both sensitivity measures assign substantial 55
Figure 3.6 Sensitivity analysis of the four-category fMRI dataset. The upper part shows the ROI-wise scores computed from SMLR classifier weights and ANOVA F-scores (limited to the 20 highest and the three lowest scoring ROIs). The lower part shows dendrograms with clusters of average category samples (computed using squared Euclidean distances) for voxels with non-zero SMLR-weights and a matching number of voxels with the highest F-scores in each ROI. 56 importance to this region. This might be due to the large inter-sample distances visualized in the corresponding dendrogram in figure 3.6. Each leaf node (in this case an average volume of two stimulation blocks) is approximately as distinct from any other leaf node, in terms of the employed distance measure, as the semantic clusters identified in the TOFC ROI. Finally, the ROI covering the anterior division of the superior temporal gyrus shows no clustering at all, and, consequently, it is among the lowest scoring ROIs of both measures. On the whole, the cluster patterns from voxels selected by SMLR weights and F-scores are very similar in terms of inter-cluster distances.
Given that these results rely on data obtained from a single participant, no far-reaching implications can be drawn. However, the distinct cluster patterns provide an indication that different levels of information encoding could be addressed in future studies. Although voxels selected in both angular gyrus and the frontal pole ROIs do not provide a discriminative signal for all four stimulus categories, they nevertheless provide some disambiguating information which the classifier was able to detect. In angular gyrus, this seems to be an animate/inanimate contrast that can also differentiate between animate stimuli belonging to two categories. Finally, in the frontal pole ROI the pattern remains unclear, but the relatively large inter-sample distances might indicate a differential code of some form that is not closely related to the investigated level of semantic stimulus categorization.
3 3 Specificity Assessment: Face vs. House Debate
To address the question of whether or not a face - specific area exists, this section presents an analysis of a decoder capable of discriminating between only face and house categories. Although a number of classifiers were tested, the analysis presented here focuses on Support Vector Machines (SVMs) [77]. As noted earlier, SVMs have many desirable properties: they can learn even in huge (>1M) feature spaces, they produce a 57 unique solution by constraining problem formulation, they generalize well with feature spaces many orders larger than the data sample size, they can be highly robust over significant levels of noise, and finally, they can learn subtle distinctions near the separating hyperplane which can disambiguate each category sample. SVM does this all without the cost of learning the complete distribution of each category (although this will turn out to be a disadvantage for visualization). To address the question of brain area identification, the whole brain was used as input 40K voxels — white matter was used as a noise background to increase the reliability of estimates). The data was then incrementally searched for voxels that uniquely identified either FACE or HOUSE stimuli. This was done exhaustively per subject per scan and cross-validated on independent sets of data. In order to increase the generalization of this approach, data from two separate object identification experiments were used. In both studies, subjects performed judgments on the two key object types (FACE, HOUSE) in two different cognitive tasks. The first task was the same as in the previous section (i.e., [25]) with a focus on the stimuli necessary for the identification of FACE and HOUSE, so that only blocks of collected for those two conditions were considered in the analysis. In the second task, memory demand is reduced by using a simple perceptual judgment in which high contrast black and white stimuli were presented in an oddball task (task 2 is designated as the OB task). In the OB task, trials consisted of a simultaneous presentation of 3 stimuli (either 3 FACEs in FACE trials or 3 HOUSEs in HOUSE trials) in different orientations with the subject being asked to identify the member of the triad that was different. In both tasks subjects achieved behavioral accuracy rates identifying objects above 80% correct. In both cases, full brain (approximately 40000-50000 voxels) data was collected, with 144 samples (77 of each type) in task 1 and 200 samples (100 of each type) in task 2. Both experiments used a block design and data provided to the analysis consisted of 7 trials in task 1 and 17 trials in task 2 (3 of the original 20 trials were eliminated to reduce autocorrelation effects). Leave-two (one 58 o eac ca ego y ocks-ou s a egy was use o assess e ge e a i a io e o ma ce C assi ica io was o e wi si g e sca s o equi a e y si g e s — c [ ] I o e o ac ie e e owes ossi e e o i ge e a i a io ecu si e ea u e e imi a io ( E was e o me (see [ 5] is a oac as e a a age o e ec i g e s eci ic o ec i e i ica io ai a eas y a es i g e mos se si i e o e s a e ai i g a oi g su seque e ai i g o is se si i e u sma e se is ocess was co i ue u i e e a e o mo e o e s e o es a e e o e was e aus i e o e e si g e sca ai o e se o a oi o e ia c oss ock co ami a io o emo y amic O sig a a e e o e acci e a y c ea i g a ge e a i a io ias sca s a e egi i g a e e o a ocks (ca ego y o es co i io s we e isca e W o e ocks we e a so ou i e y e ou a es e agai s si g e s om ose ocks ( o i g ou a o e s i a ock u i sam e a e is a oac e uce a y u wa ias om ossi e co e a io wi i ocks A seco es ock was a so use i o e e ime s o u e e e a y em o a co ami a io o co usio e wee ca ego y es o ses I o e o e S M c assi ie o o e a e o e a a aw o e a a a o e co e e so a a ues e wi i [-1 +1] a ge wo ossi e co e sio sc emes we e es e sca e e ce age c a ge e a i e o ase i e a -sco es e a i e o e ase i e ( es co i io ocks S Ms ai e o -sco es o i e e e ge e a i a io o c ose es cases a so ese we e c ose o u e a a ysis Eac 3 sca ( ai o e s o y co ai i g oug y u o o e s was use as a i u sam e o e c assi ica io i a i io o a a e o e s imu us co i io ( ACE o OUSE ese e u i g a sca Su ec a a was su mi e o a so -ma gi S M wi a a e age e s e ackwa ea u e e imi a io o 1 -5 o e s (a o ima e y 1 % o 15% e o e ia emo a a e u i 1 o e s was eac e a e emo a o o e o e a a ime u i o y o e o e was e A eac s e a o e s se si i i y wi i ai e S M was use as a asis 59
bl 3 1 Minimal Error and Associated Voxels Count.
for keeping or eliminating that voxel. Voxels with the smallest weights on each step were eliminated.
In order to obtain an unbiased estimate of SVM generalization performance each dataset was split into training and testing datasets. Similar to [26], a N-l block bootstrap procedure was implemented. Specifically, for each training set a single block from each category (FACE or HOUSE) was taken out for testing, leaving B-l per category used for training. All possible combinations of testing blocks from the two categories were taken, making a total of 144 NB (12x12 blocks) and 100 OB (10xl0 blocks) training/testing
datasets . Each 'training/testing dataset proceeded through RFE independently and performance was averaged over all classifiers/bootstraps to obtain a generalization estimate for a given subject/SVM. Shown in Figure 3.7 is the average error of the generalization as a function of the voxels remaining in the training set. Each line shows a single subject's performance on out of sample pairs of HOUSE and FACE exemplars.
The color of the line indicates the group of subjects either in NB (red) or OB (green) tasks, note that near the minimum they significantly overlap, with the oddball task
starting at a lower error on initial learning 9 .
8For two subjects, one in the OB task and the other in NB, a single block in each case was found to be corrupted, and for those cases the number of bootstrap opportunities were 81 and 121 respectively. 9 One possible reason for the advantage in the OB task could simply be the difference in scanner strength which was 1.5T for the NB task and 3T for OB task. Another possible explanation is that the OB task actually required a category judgment, while the NB task required only a simple stimulus identification. This would change the behavioral contrast between stimuli. In either case, a parametric experiment that varied the difficulty of the categorization judgment might resolve this intercept difference. 60
In table 3.1 is shown the exact minimum error and associated voxels remaining at that point. The minimum average error for all subjects in both tasks indicated nearly 98% correct on out of sample cases with 3 subjects in the NB task and 2 from the OB task showing ZERO generalization error.
Brain Areas Identification Which areas of the brain are uniquely required for the identification of FACEs and HOUSEs? In a fashion similar to that described in previous sections, constructed classifiers were examined by doing a sensitivity analysis that effectively asked what a voxel's contribution to the classification error was. If category error significantly changes when the voxel is removed, then it can be inferred that this voxel is contributing in a proportionally diagnostic way to the identification of the FACE or HOUSE category.
All brain volumes (tasks, subjects) were first registered using the following steps: (l) a subject's sample bold scan, anatomical, and MNI anatomical were skull stripped using BET from FSL tools [76], (2) the stripped anatomical was co-registered to the stripped MNI using FLIRT (from FSL) to obtain an anatomical to MNI transformation,
(3) the stripped anatomical was co-registered to the stripped BOLD using FLIRT to obtain an anatomical to BOLD transformation, and finally (4) the anatomical was transformed into BOLD space using anatomical to BOLD transformation for easy visualization of activation patterns.
In order to localize the FFA and PPA for analysis, the standard method in the field for localizing category responsive voxels was used. This involved first finding voxels that significantly responded to FACE > "other non-FACE objects" or HOUSE> "other non-HOUSE objects". In the case of the NB task, the original FACE and HOUSE masks in Haxby et al's [25] study were used. Those masks ranged from 20 voxels to about 100 voxels. In the case of the oddball study, independent localizer scans were used in standard GLM contrasts for FACE>HOUSE and HOUSE>FACE. Although seemingly 6
G n r l z t n Err r th t r El n t n
0 00 000 0000
x l n n
igu e 3 7 Generalization error for all ten subjects on single scans for both kinds of stimuli held out from a training exemplars. Recursive voxel elimination produced a minimum for half the subjects at zero error, while over all subjects on single scans, across both tasks there is nearly a 98% correct generalization on unseen scans.
au o ogous is is o e e ess e s a a oce u e wi i e i e a u e o es a is se ec i e OIs o su seque es i g ece y e e as ee co si e a e e a e a ou is me o [7 79] u o e e ess is oce u e emai s e ima y way o e e mi i g candidate o e s o mo e iag os ic a oac es I e o owu a a ysis A a A masks we e i e sec e wi e ca cu a e se si i i y masks o e e mi e e amou o o e o e a e i icu y i i e i yi g e A i a icu a is a e e ca e co si e a e a ia i i y i e si e o masks ac oss su ec s I e o igi a a e [57] a a gue o e e is e ce A o y 1 o e 1 su ec s ac ua y a A ac i i y es i e i s imi a o s i e i ica io o e A y G M co as s o oca i e sca s e ec s e s a e o e a i e eu oimagi g ie o se ec i g ca i a e o e s
Se si i i y A a ysis e e a e a um e o ossi i i ies o i e i yi g iag os ic ai a eas A o ious c oice mig e o use a o e su o ec o s emse es owe e usi g ec o s a a e s ic y i e ma gi may e i a o ia e i c a ac e i i g e iag os ic ai a eas o OUSE a ACE i asmuc as ey mig e ese ai o umes a a e ea e se a a i g su ace a e e o e u e ese a i e o e ai es o se o ei e OUSE o ACE s imu i e o e ossi i i y is e O -su o ec o s ( S esc i e y aCo e e a [31] ese a e ec o s a a e is i u e eyo e su o ec o s a a oug some may e o o y ica o e ai es o se u o u a e y ma y wi o e I ac i ge e a e e assumi g a Gaussia s ea o e S s o y a sma mi o i y wi e y ica o " es " mem e s o e ca ego y ecause S M o imi es a ma gi e wee e wo ca ego ies e ac ua is i u io o mem e s o e ca ego ies is e ec i e y ig o e e ce i o ica y e same o e y a makes S M a e ce e ca i a e o c assi ica io i ig ime sio s is e o e a a so makes i icky o isua y i e e o is easo a isua i a io a oac e wee ese wo e eme ossi i i ies was c ose E ec i e y i im eme s a se si i i y/ e u a io a oac (e.g., [ ] w ic 3 measures the error for a given category (FACE, HOUSE) when the voxel is present verses when it is removed. Specifically, to estimate SVM-based sensitivity, we used one of the simplest criterion that have been proposed [41, 65, 80], which is simply the reciprocal of the separating margin width W = 11/0112, where w Σi Minimization of this criterion leads to maximization of the margin width. In the case of
Linear SVM, the squared values of the separating plane normal coefficients (i.e. q as stated, effectively correspond to the change of the criteria W as if the voxel i is removed. Therefore, the classifier is less sensitive to the features with low 7 During recursive feature elimination we sequentially eliminated features with the smallest w,?. Additionally, in order to increase diagnostic selectivity we derived weights for the FACE category by using only FACE SVs and for the HOUSE category by using only HOUSE category SVs. Thus, higher voxel values tended toward typical regions in the classification space for the SV appropriate category. We will refer to these direction
selective voxel coefficients as diagnosticity of the voxel direction for or against HOUSE (in blue) and FACE (in red). Shown in Figure 3.8 are diagnosticity measures for two typical subjects (one from NB and one from OB). A non-parametric method for thresholding these diagnosticity distributions at p<0.01 was used for each subject, taking into account the non-Gaussian aspect of the underlying diagnosticity distributions (c.f. with Type 4 method constructing empirical cumulative distributions; linear interpolation of the cdf [81]). Slices are shown for each subject with specific voxel clusters. Note that SVM finds fairly contiguous regions, despite the specific bias not to. In fact, at lower thresholds (0.05) the diagnosticities begin to fractionate through the whole brain. Unlike
a regression analysis (e.g., GLM) which finds "selective" or detection areas, these voxel patterns are unique identification areas in that they are contributing to a correct classification of one category against the other, and moreover, they have been cross-validated in independent data sets so that the likelihood is high that they will generalize to other unseen cases of either FACEs or HOUSEs.
ACE IAG OUSE IAG ACE IAG OUSE IAG
igu e 3 Voxel sensitivities for brain areas that are diagnostic for FACE (red and on left of each panel) and for OUSE (blue and right of each panel). In the top left corner we show FFA (fusiform face area as measured by a localizes task) in the first paired panel (same subject, same slice, same task) which shows subject NB l. Below that is subject OB3 also showing FFA in both slices. On the left bottom panel we show PCC which is diagnostic for HOUSE and FACE which both show sensitivity (NB2). The next three paired images show distinctive areas between diagnostic areas of FACE and HOUSE. The next panel above on the right shows pSTS (posterior superior temporal sulcus) which is diagnostic for FACE against HOUSE (subject OBl), the next panel below shows another FACE diagnostic areas (MFG BA9, subject OB5), and finally in the last paired panel on the right we show a HOUSE diagnostic area (LO) for subject NB5. 65
e ques io s ose a e egi i g o is sec io may ow e a esse y assayi g e a o e es o i e i ica io a eas i o e o e e mi e w e e ( e e a e u ique i e i ica io a eas o ACE a OUSE a ( w e e e A o A a e ca yi g o e isc imi a i e i o ma io a ou o e o e e e s imu i (i is case ei e OUSE o ACE es ec i e y
Brain Areas Identifying FACEs and HOUSEs igu e 3 9 s ows a a eas a es e a we e commo (i e sec io se ¹ o a su ec s i o asks a we e ei e ig y iag os ic o ACEs ( a o o OUSEs ( ase o e se si i i y/ iag os ici y a a ysis as esc i e e o e e a o s ows e e ce o o e s associa e wi eac a ea a e 1 es o usi g e o a ame ic me o s e o a i eac a ca e com u e y mu i yi g e e ce age agai s e o a um e o se si i i ies a o e es o i eac ca ego y ( ACE=3 3 a OUSE=35 so o e am e a ou a e o e s i o OUSE a ACE a ou 15 eac a i o e A o e a ese a eas a e ase o e es c oss- a i a e si g e sca c assi ie s a we e ea y 9 % co ec o ou -o -sam e e em a s e ce e a eas i e i ie u e ese co s ai s a e o ase o e usua "o ec se ec i e" i e e a io a e e o e o su ec o e esu a am igui y wi me o s a a e ase o simi a i y o associa io ey a e i ac a ei i a o a i is ic se se ecessa y a su icie o i e i ica io o ACEs o OUSEs I is im o a a is oi o c a i y a e ese c assi ie s cou a e ou si g e a eas o si g e o e s as e ic i e o a si g e ca ego y I ac i ea me o s e o e iase owa s usi g si g e ime sio s ( o e s i is case o mi imi e c assi ie e o es ecia y i a ea co e a io s e o e
¹ °A very conservative harvesting was used to include only voxels that were in all ten subjects voxels that were above p<0.01 therefore appeared in both tasks. Also, sensitivities were initially averaged over all generalization runs in order to increase the sample power of the voxels sets per subject. In any case, there was significant overlap (64%) of the same voxels across generalization runs and this tended to covary with the minimum generalization error reached for that classifier. It is appropriate to average across the runs, since any differences in voxel sets are due to sampling error in classifier estimation and data noise. sma e wee ea u es o o e s I e e a e a ge co e a io s e wee o e s ese cou e mi imi e y usi g a eu a e wo k w ic ca eco e a e ea u es as i c assi ies ea es eig o c assi ie s suc as use y a y e a [ 5] i ac a e mo e iase owa s se s o ea u es si ce ei simi a i y i c eases i c eme a y wi mo e commo o e a o o e s Co seque y i is ossi e o egi a swe i g e ques io ose i is sec io A e e e o e a eas o e ai a a e iag os ic o e i e i ica io o ACEs o OUSEs? o o e a eas o e ai ca y isc imi a i e i o ma io o e a A a A? C ea y om e ese a a ysis e a swe is "yes" O e a e iag os ic o i e o e e a a eas o o ACEs a OUSEs a e somew a simi a u o i e ica es i e e a ea a ce o is i c a eas e wee ca ego ies ese a eas a so seem o e a o a a ge e wo k I ac igu e 3 s ows a se ec io o e ese a i e e am es om i e e su ec s ac oss o asks (O a I e igu e eac ai e se o igu es i a ow is e same su ec s owi g e iag os ici ies o ACE i e (o e e o eac ai a OUSE i ue (o e ig o eac ai I e i s wo ai e se s we s ow A se si i i y i wo i e e su ec s ac oss e wo i e e asks i o ACE a OUSE s imu us ese a io s o a su ec s usi o m gy us (a A masks o e a e wi 9 % o e G o e s o a su ec s a o asks a ea e o o ACE a OUSE i ica i g iag os ic a ue o is a ea a was ei e s ecia i e o u ique A so o a su ec s ac oss e asks e a a i ocam a gy us (a A masks o e a e wi mo e a 7 % a e os e io ci gu a e co e ( CC we e a so iag os ic o ACE a OUSE e e ee se s s ow distinctive iag os ic a eas i c u i g u ique S S se si i i y o ACE u o OUSE O e commo iag os ic a eas i c u e mi e occi i a gy us (i c u i g a ea O a mi e o a gy us ( A 9 I ge e a a e wo k o a eas was i e i ie as iag os ic o ese wo s imu us y es wi ACE a i g a e o a a ea i o e w i e OUSE a ea e o i o e a a ea k ow o isua s a e/ e u e ocessi g 67
x l Gr p n t f r ACE, = 6
x l Gr p n t f r OUSE, = 8
Figure 3.9 Voxel area cluster relative frequencies intersected over all subjects and both tasks. For FACE diagnosticities (top), six areas were identified at p<0.01: Fusiform Gyrus (FG), Parahippocampal Gyrus (PHG), Middle Frontal Gyrus (MFG), Superior Temporal Sulcus (STS), and Precuneus (PCUN), with relative frequencies based total number of voxels (N=363) over all areas. For HOUSE diagnosticities (bottom) there were five areas, (at p<0.01, N=358) some the same, some different, including FG, Parahippocampal Gyrus (PHG), Posterior Cingulate Cortex (PCC), PCUN, and Lateral Occipital (LO). Note that overlapping FACE and HOUSE areas, include FG, PHG PCUN and PCC. Distinctive areas for FACE included STS, MFG, while for HOUSE distinctive areas included only LO. 68
o e a e c aim is o a ACE s imu i o o equi e s ecia i e u c io s (e.g., s a e/ e u e ocessi g a e i seems ike y a e a eas a a e ee i e i ie i s ecia i e e e ime s i uce us o use a e s a a e some e ica ami ia i y wi e ese e s imu i a w ic may e mis ea i g i o e co e s w e e ose ai a eas a e i e ac i g wi ma y o e ai a eas is a e i g o em a o e im ica io s om is s u y a e iscusse i e o owi g sec io
3.4.4 Discussion
is sec io o e isse a io a em s o a swe a sim e ques io a as ee agui g e o ec ecog i io ie o e as 1 yea s is e e a u ique a ea o e ai a is use o y o i e i y aces? u e is e e a u ique a ea o e ai w o s so e u ose is o i e i y ouses? ase o e ese a a ysis e a swe is " o" o e ai is s a eme ca e qua i ie i wo ways i s a c aim is o ei g ma e a ou some ge e a o e y o "com i a o ia " co i g oug ou e ai u a e e esu s o e a a ysis ese e e e a e s eci ic o e i e io em o a o e a o o ec ecog i io ocesses i a icu a Seco a i e eso u io o a a (see iscussio e ow mig c a ge e esu s ama ica y as mo e e ai wi i i e io em o a o e a A i a icu a a e ma e a ai a e o e e ess e a a ysis i i a is oi e wo k o ai a eas a a e iag os ic o ei e ACE s imu i o o OUSE s imu i is is co sis e wi a y e a s [ ] mo e o ace e ce io w ic i s e same a eas o e i o e i a commo e wo k o ocessi g aces e commo a eas i is a a ysis i c u e e usi o m ace a ea s eci ic a eas o e i gua gy us (w ic s owe u weak y i is a a ysis e mi e occi i a gy us ( O a ecu eus is i c i e a eas o ACE s imu i i c u e S S a mi e o a gy us ( A9 is i c i e a eas o OUSE s imu i i c u e o y a ea O owe e o ACEs e e seems o e o e i e ce i e ese a a ysis a e A 69 is either unique diagnostically nor specialized insofar as it was identified in every subject responding to the HOUSE stimuli. So this observation would seem to put a rest to the controversy that there are unique areas of the brain that only respond to specific tokens or types. But not exactly. The claim of Kanwisher [60] is complicated by the fact that many areas of the brain seem to be required for identification of these FACE and HOUSE stimuli and the assumption that areas that are a ea y selected through an independent localizes (see earlier discussion) tests are uniquely and solely involved in identifying these object types. So to paraphrase Kanwisher, does the FFA and PPA provide discriminative information about other object types other than FACE and HOUSE respectively? The answer based on the present classifiers is "yes". Since most of the voxels identified for either FACE or HOUSE were squarely sitting in the FFA of each subject, this would seem to be definitive evidence that the FFA does involve discriminative information about object types other than FACEs, that is, in this case, HOUSEs. We also must note that the exact overlap of the areas between kg FFA-FACE and FFA-HOUSE are not exactly the same, despite voxels that are squarely in the same place (see crosshairs), nonetheless this is normative in the field as
the FFA will have different locations and shapes across subjects and even within a subject across sessions or experiments will vary in strength, location and shape. Given HOUSEs and FACEs do look different, it is not impossible that the FFA does code these stimuli differently, and a an experiment using fMRI data obtained at a higher resolution might very well produce different distributions of recruited voxels in the FFA for each stimulus, which might be more consistent with Kanwisher's claims. Nonetheless within the state of the art, our localizations of diagnostic cortical areas have no more variability or lack of precision than that which appears in the standard literature. A potentially more difficult issue for Kanwisher is that in our results there were also brain areas that were distinctive for FACE or HOUSE other than FFA or PPA. For example, it would be possible to argue that pSTS is the superior temporal sulcus FACE area, the pSTSFA! We know that this 70 area is sensitive to "biological motion", why could there not be a part of it specifically dedicated to the unique identification of FACEs? Other category tests would be very likely to show these areas area not distinctive, but more likely part of a larger network of some kind of FACE identification system. In terms of uniqueness and given the specificity of the claim, that the FFA is claimed to be unique and necessary for FACE identification, then if it were also to be diagnostic for any other category such as HOUSEs, the claim must be refuted. Again, this would seem to lay the matter to rest, there is no FACE area per se, at least in the way that Kanwisher has defined it. Clearly, it is obvious to conclude that there is no area that responds uniquely and solely to FACEs, that could be found in a whole brain assay of a high performance single TR classifier. C A E
MU IMO A A A YSIS O M I A EEG A A
e e is a i c easi g um e o e o e E/MEG/ M I co oi s u ies w ic a em o gai e a a ages o a mu imo a a a ysis o e e ime s i o i g e ce ua a cog i i e ocesses isua e ce io [ 3- 7] a mo o ac i a io [ 3] es o se a ici a io ( ia co i ge ega i e a ia io C [ ] soma ose so y ma i g [ 9 9 ] M I co e a es o EEG y ms [91-95] a ousa a a e io i e ac io [9 ] au i o y o a asks [97-99] assi e eque cy o a [1 ] i uso y igu es i isua o a asks [1 1] e ce ua c osu e [1 ] a ge e ec io [1 3 1 ] ace e ce io [1 5] s ee [1 ] a guage asks [ 1 7 1 ] se - egu a io o EEG- ase CI [1 9] a e i e sy [11 -1 ] esea c e s w o a oac mu imo a a a co ec io a a a ysis u i o a mu i u e o o ems wi ega o e acquisi io a a a ysis o eu a a a E am es o ese o ems wi e o i e i e i s sec io o is c a e as we as a iscussio o e a a ages a e is i g mu imo a a a ysis ec iques o e o e u imo a a a ysis a oac es I wi e s ow ow e e o a io o ew ec iques o a a y i g com o e s o E/MEG sig a s i a i io o e use o ose a ea y i e is e ce ca e use success u y i co oi a a ysis us is isse a io ou i es e o ems wi e is i g me o o ogies a o e s a o e mu imo a a a a a ysis a oac as a so u io
71 72
4.1 Multimodal Experiment Practices
When you build bridges you can keep crossing them
— Rick Pitino
Although a strong static magnetic field itself seems to have no effect on the brain electric activity (see [121, chap. 4.2] for review), obtaining non-corrupted simultaneous recordings of EEG and fMRI is a difficult task due to interference between the strong MR field and the EEG acquisition system. Because of this limitation, a concurrent
EEG/fMRI experiment requires specialized design and preprocessing techniques to prepare the data for the analysis. The instrumental approaches described in this section are specific to collecting concurrent EEG and fMRI data. For obvious reasons MEG and fMRI data must be acquired separately in two sessions. However, even when MR and MEG are used sequentially, there is a possibility of contamination from the magnetization of metallic implants which can potentially disturb MEG acquisition if it is performed shortly after the MR experiment. A series of validation studies [122] has been carried out to prove the viability of collecting EEG concurrently with fMRI. These studies use existing design schemes and post-processing to isolate the contaminated EEG signal. Thus the study [122]
showed high concordance in signal characteristics between various neuronal activations (steady state visual evoked potentials (SSVEP), lateralized readiness potentials (LRP), and frontal theta) captured with EEG in concurrent with fMRI or separate sessions. ROC analysis [123] of Distributed Equivalent Current Dipole (DECD) maps of VEP obtained from EEG in the magnet with and without active fMRI acquisition, together with DECD EEG maps obtained outside of the magnet [124] showed the effectiveness of the removal of artifacts inherent in fMRI-EEG concurrent acquisition. 73
4.1.1 Measuring EEG During MRI: Challenges and Approaches
Developing methods for the integrative analysis of EEG and fMRI data is difficult for several reasons, not least of which is the concurrent acquisition of EEG and fMRI itself which itself has proved challenging (see [125] for recent overview). The nature of the problem is expressed by Faraday's law of induction: a time varying magnetic field in a wire loop induces an electromotive force (EMF) proportional in strength to the area of the wire loop and to the rate of change of the magnetic field component orthogonal to the area. When EEG electrodes are placed in a strong ambient magnetic field resulting in the EMF effect, several undesirable complications arise:
• Rapidly changing MR gradient fields and RF pulses induce voltages in the EEG leads placed inside the MR scanner. Introduced potentials greatly obscure the EEG signal [126]: higher gradients magnetic field strengthen the artifacts introduced into the EEG signal [127]. This type of artifact is a real concern for concurrent EEG/MRI acquisition. Due to the deterministic nature of MR interference, hardware and algorithmic solutions may be able to unmask the EEG signal from MR disturbances. For example, Allen et al. [128] suggested an average waveform subtraction method to remove MR artifacts which is effective in case of deterministic generative process of a signal [129]. This type of method can be
applied even with higher magnetic fields. A study with both phantom and animal VEP that used a 4.7 T [127], found that up to 90% of the amplitude of EEG signal acquired outside of the magnet could be restored without introducing a temporal offset. However, it is important to note that time variations of the MR artifact waveform can reduce the success of this method [91, 130]. The problem can be resolved through hardware modification that increases the precision of the synchronization of MR and EEG systems (e.g., Stepping Stone Sampling [131]), or during post-processing by using precise timings of the MR pulses during EEG 74
wa e o m a e agi g [91 13 ] A go i mic a ig me o M I a EEG wa e o ms wi su -sam e ecisio a e ee use success u y [133] Acquisi io o a i ea a i ac wa e o m is a so ossi e ia a i io a wi e oo s a e sca O e ec iques a a e ee o ose o e uce M g a ie a a is oca iog a ic a i ac s a ge om ec ica a oac es suc as i o a e ec o es i a wis e co igu a io [13 ] o e use o os - ocessi g ec iques suc as s ec a omai i e i g s a ia a acia i e i g CA [135-137] igu e ICA [see 135 13 -1 1] a e e a mu is age ocessi g i e i e [1 ]
• E e a s ig mo io o e EEG e ec o es wi i e s o g s a ic ie o e mag e ca i uce sig i ica EM [1 3 1 ] o i s a ce a i e u sa i e mo io e a e o a ea ea yie s a a is oca iog a ic a i ac i e EEG a ca e oug y e same mag i u e as e EEG sig a s emse es [1 13 ] Usua y suc a i ac s a e emo e y e same a e age wa e o m su ac io
ecom osi io ( . ., [13 1 5] mu i e sou ce co ec io (MSC [1 ] o c us e i g [133] me o s O e a wa e so u io a as ee sugges e o e e uc io o mo io e ec s is o use ua wis e ec o es w e e co secu i e wis s i o osi e i ec io s o e ow ca ce ou e e ec s o e mo io o g a ie swi c i g [13 ] o e uce e a is oca iog a ic a i ac e ec o es a e i m y a age o e su ec s ea [1 1]
• I uce e ec ic cu e s ca ea u e e ec o e ea s o a ai u o e e o e ia y a ge ous e e s [1 7] as ic e ec o es wi i aye s o si e c o i e o go a oys a e use o o i e su icie im e a ce o e o age ea i g o e sca wi mi ima im ac o e qua i y o e M sig a [1 ]
Cu e - imi i g e ec ic com o e s ( esis o s E a sis o s t . a e usua y ecessa y o e e e e e o me o uisa ce cu e s w ic ca a e i ec 75
igu e 1 EEG M artifact removal using PCA. EEG taken inside the magnet (top); EEG after PCA-based artifact removal but with ballistocardiographic artifacts present (center); EEG with all artifacts removed (bottom). After artifact removal it can be seen that the subject closed his eyes at time 75.9 s. (Courtesy of M. Negishi and colleagues, Yale University School of Medicine) 7
co ac wi su ec s sca Simu a io s ca o i e e sa e owe a ge a s ou e use o a icu a co igu a io s o coi / owe /se so s i o e o com y wi A gui e i es [1 9]
A o e co ce is e im ac o EEG e ec o es o e qua i y o M images e i o uc io o EEG equi me i o e sca e ca o e ia y is u e omoge ei y o e mag e ic ie a is o e esu i g M images [ 3 1 ] ece i es iga io s s ow a suc a i ac s ca e e ec i e y a oi e [1 ] y usi g s ecia y esig e EEG equi me [13 ] s ecia i e geome ies a ew "M -sa e" ma e ia s (ca o i e as ic o e ea s o es e i ue ce o a gi e EEG sys em o M I a a a com a iso o e a a co ec e o wi a wi ou e EEG sys em ei g ese s ou e co uc e A a ysis o a a usua y emo s a es e same ac i a io a e s i wo co i io s [ 3] a oug a ge e a ec ease i M I S is o se e w e EEG is ese i e mag e A co ec io o e a e ec ¹ (w ic a e use o o wa E/MEG mo e i g i s e o owi g i s -o e co ec io o e eg igi e
σ = 1 1 - o- o B = 1 5 [15 ]
4.1.2 Experimental Design Limitations
e e a e wo ways o a oi i g e i icu ies associa e wi co ec i g EEG a a i e mag e ( co ec EEG a MRI a a se a a e y o ( use a e e ime a a a igm a ca wo k a ou e o e ia co ami a io e wee e wo mo a i ies e c oice e wee ese wo a e a i es wi e e o e co s ai s associa e wi esea c goa s a me o o ogy o e am e i a e e ime ca e e ea e mo e a o ce wi a ig eg ee o e ia i i y o e a a se a a e E/MEG a M I acquisi io may e a o ia e [9 97 1 3 1 5] I cases w e simu a eous measu eme s a e esse ia
¹ The Hall effect is the production of a potential difference (the Hall voltage) across an electrical conductor, transverse to an electric current in the conductor and a magnetic field perpendicular to the current (http://en.wikipedia.org/wiki/Hall_effect). 77 for the experimental objectives (e.g., cognitive experiments where a subject's state might influence the results as in monitoring of spontaneous activity or sleep state changes), one of the following protocols can be chosen:
Triggered fMRI: detected EEG activity of interest (epileptic discharge, etc.) triggers MRI acquisition [110, 111, 114, 151]. Due to the slowness of the HR, relevant changes in the BOLD signal can be registered 4-8 s after the event. The EEG
signal can settle quickly after the end of the previous MRI block [134], so it can be acquired without artifacts caused by RF pulses or gradient fields that are present only during the MRI acquisition block. Note that ballistocardiographic and motion-caused artifacts still can be present and will require post-processing in order to be eliminated. Although this is an elegant solution and has been used with some success in the localization of epileptic seizures, this protocol does have some drawbacks. Specifically, it imposes a limitation on the amount of subsequent EEG activity that can be monitored if the EEG high-pass filters do not settle down soon after the MR sequence is terminated [106]. In this case, EEG hardware that does not have a long relaxation period must be used. Another drawback with this approach is that it requires online EEG signal monitoring to trigger the fMRI acquisition in case of spontaneous activity. Often experiments of this kind are
called EEG-correlated fMRI due to the fact that offline fMRI data time analysis implicitly uses EEG triggers as the event onsets [129];
Interleaved EEG/fMRI: the experiment protocol consists of time blocks and only a single modality is acquired during each time-block [95, 150]. This means that every stimulus has to be presented at least once per modality. To analyze ERP and fMRI activations, the triggered fMRI protocol can be used with every stimulus presentation so that EEG and MR are sequentially acquired in order to capture a clean E/MEG signal followed by the delayed HR [85]; 78
Simultaneous fMRI/EEG: e- ocessi g o e EEG sig a me io e i Sec io 1 is use o emo e e M -cause a i ac s a o o ai a es ima e o e ue EEG sig a owe e ei e o e e is i g a i ac emo a me o s as o e o e ge e a e oug o wo k i e e y y e o EEG e e ime a a a ysis I is es ecia y i icu o use suc a acquisi io sc eme o cog i i e e e ime s i w ic e EEG e oke es o ses o i e es ca e o sma am i u e a com e e y o e w e me y e M oise [15 ]
4.2 Forward Modeling of BOLD Signal
E e i e case o a i ac - ee EEG a M I a a acquisi io e success u a a ysis o a a co ec e i a mu imo a e e ime emai s o ema ic e mai o em o mu imo a a a ysis is e a se ce o a ge e a u i yi g accou o e O M I sig a i e ms o e c a ac e is ics o eu o a es o se a ious mo e s a e ee sugges e i c u i g coa se mo e i g o O sig a i e co e o a i ea ime I a ia Sys em ( IS as we as ge e a mo e s o e O sig a i e ms o e ai e io ysica ocesses (M a oo sys em c a ac e is ics [153] Balloon [15 ] (i s i ea i e [155] a ge e a i e [15 ] o ms o Vein and Capillary [157] mo e s e sim e mo e s a e o ge e a e oug o e ai e a ia i i y o e O sig a w e eas com e a ame ic mo e s a e y ea i y o io k ow e ge o uisa ce a ame e s ( ue o io ysica e ai s a mos e e a e a e ia e a s aig o wa mea s o es ima io is ac makes i u ike y a suc com e e si e mo e s ca e use as e ia e ge e a i e mo e s o e O sig a Cu e y esea c is o goi g i e a em o i e i y s a e uisa ce a ame e s [15 ] a o e i e o e mo e s sui a e o a a o ai e i i e e mo a i ies A i e es i g heuristic model o eu o a ac i a io a i s i ue ce o O a EEG sig a s was ece y sugges e y Ki e e a [159] is mo e e a es O sig a o 79 the changes in spectral characteristics of the EEG signal during activation. The proposed model formulation is consistent with the results of a number of multimodal experiments that used other forms of analysis. Thus, this model seems promising in being able to reveal reliable interdependencies between different brain imaging modalities. The following section describes modeling issues in greater detail to further underscore the limited applicability of many multimodal analysis methods covered in Section 4.3.
1 C nv l t n l M d l f O S n l
A common paradigm in early studies that collected fMRI data used simple contrast designs (e.g., block design) to exploit the assumed linearity between design parameters and the HR. The belief underlying the use of block designs is that they can amplify the
SNR because the HR possesses more temporal resolution than indicated by the scan acquisition time (TR).
In order to account for the present autocorrelation of the HR caused by its temporal dispersive nature, Friston et al. [160] suggested modeling HR with an LTIS, so that the HR is modeled by convolving an input (joint intrinsic and evoked neuronal activity) q(t) with a hemodynamic response function (HRF) h(t).
Because localized neuronal activity itself is not directly available through non- invasive imaging, verification of LTIS modeling on real data, as a function of parameters of the presented stimuli (i.e., duration, contrast), is appropriate. The convolutional model was used on real data to demonstrate linearity between the BOLD response and the parameters of presented stimuli [161, 162]. In fact, many experimenters have shown apparent agreement between LTIS modeling and real data, as well as its superiority over more complex, non-linear models when dealing with noisy data
[15 ] S eci ica y i as ee ossi e o mo e es o ses o o ge s imu us u a io s y co s uc i g em usi g e es o ses o s o e u a io s imu i a ge esu s co sis e wi IS mo e i g ecause o i s e ic i e success i s e a i e ease o use a i s i e e e ce om io ysica e ai s is mo e i g a oac as ecome wi e y acce e O e ow si e IS as a mo e i g co s ai is e y weak a co seque y a ows e use o make a i a y c oices o a ame ic ase so e y o e e e ce a ami ia i y O e e yea s mu i e mo e s o e a e ee sugges e e mos o u a a wi e y use u i ow is a si g e o a i i y e si y u c io ( o e Gamma is i u io y [1 3]a I was e e e y G o e [1 ] o e o m eco o u io o e sig a uisa ce a ame e s ( 1 1 a o e e we e es ima e o mo o a au i o y a eas
as e sum o wo u sea e s o e Gamma is i u io e i s e m ca u es e osi i e O a e seco e m ca u es e o e s oo o e o se e i e O sig a Ma y o e mo e s o o sim e a so is ica e a e ee sugges e ese i c u e oisso [1 ] Gaussia s [1 5] ayesia e i a io s [1 -1 ] a eco o u io [1 9] amo g o e s e a icu a c oice o a gi e mo e is o e mo i a e y some ac o s o e a ose a isi g om io- ysics; i.e., easy ou ie a s o ma io e ese ce o a os - es o se i o " es - i " o e ies Mo e i g o e oug eco o u io o e sig a as ee s ow o im o e e ec a i i y o co e io a G M a a ysis o e am e w e a ie o e i e ic a a [119] is is o su isi g gi e e c a ac e is ic a ia i i y o e O sig a a i u a e o a ious uisa ce a ame e s Mo e o is o ic wi e iscusse i e e sec io (Sec io 81
Following the development of the convolutional model to describe BOLD response, the issue of HR linearity became an actively debated question. If HR is linear, then with what features of the stimulus ( . ., duration, intensity) or neuronal activation
( . ., firing frequency, field potentials, frequency power) does it vary linearly? As a first approximation, we can define the range of possible values for those parameters in which HR was found to behave linearly. For example, early linearity tests [164] showed how difficult it was to predict long duration stimulus effects based on an estimated HR from shorter duration stimuli. [170] reviewed existing papers describing different aspects of non-linearity in BOLD HR and attempted to determine the range of values associated with linearity in three cortical areas: motor, visual and auditory complex. The results of these analyses have shown that although there is a strong non-linearity observed for small stimulus durations, long stimulus durations show a higher degree of linearity. It appears that a simple convolutional model generally is not capable of describing the BOLD responses in terms of the experimental design parameters if such are varying in a wide range during the experiment. Nevertheless LTIS might be more appropriate to model BOLD response in terms of neuronal activation if most of the non-linearity in the experimental design can be explained by the non-linearity of the neuronal activation itself.
4.2.2 Neurophysiologic Constraints
In the previous section the subject of linearity between the experimental design parameters and the observed BOLD signal was explored. For the purpose of this work it is more relevant to explore the relation between neuronal activity and HR.
It is known that E/MEG signals are produced by large-scale synchronous neuronal activity, whereas the nature of the BOLD signal is not clearly understood. The BOLD signal does not correspond to the neural activity that consumes the most energy [171], as early researchers believed. Furthermore, the transformation between the electrophysiological indicators of neuronal activity and the BOLD signal cannot be linear for the entire dynamic range, under all experimental conditions and across all the brain areas. Generally, a transformation function cannot be linear since the BOLD signal is driven by a number of "nuisance" physiologic processes such as cerebral metabolic oxygen consumption (CMRO ), cerebral blood flow (CBF) and cerebral blood volume (CBV) as suggested by the Balloon model [154], which are not generally linear. Due to the indirect nature of the BOLD signal as a tool to measure neuronal activity, in many multimodal experiments a preliminary comparative study is done first in order to assess the localization disagreement across different modalities. Spatial displacement is often found to be very consistent across multiple runs or experiments (see Section 4.3.3 for an example). Specifically, observed differences can potentially be caused by the variability in the cell types and neuronal activities producing each particular signal of interest Nunez and Silberstein [172]. That is why it is important first to discover the types of neuronal activations that are primary sources of the BOLD
signal. Some progress on this issue has been made. A series of papers generated by a project to cast light on the relationship between the BOLD signal and neurophysiology, have argued that local field potentials (LFP) serve a primary role in predicting BOLD signal [173, and references 27, 29, 54, 55 and 81 therein]. This work countered the common belief that spiking activity was the source of the BOLD signal [for example
174] by demonstrating a closer relation of the observed visually evoked HR to the local field potentials (LFP) of neurons than to the spiking activity. This result places most of the reported non-linearity between experimental design and observed HR into the non-linearity of the neural response, which would benefit a multimodal analysis. Simultaneous EEG and fMRI recordings during visual stimulation [175] supported the idea that non-linearity of BOLD signal primarily reflects non-linearity of neuronal response itself by demonstrating linearity between principal current density from EEG and the estimated value of the synaptic efficacy in the modified balloon model [176]. 83
To validate pioneering research of Logothetis et al. [177] (see Section 4.2.2) spiking activity was correlated with BOLD response across multiple sites in the cat visual cortex Kim et al. [178]. The correlation varied from point to point on the cortical surface and was generally valid only when data were averaged at least over 4-5 mm spatial scale, once again questioning spatial specificity of BOLD response.
Note that the extracellular recordings of the experiments described above, were carried out over a small ROIs, therefore they inherit the parameters of underlying hemodynamic processes for the given limited area. Thus, even if LFP is taken as the primary electrophysiological indicator of the neuronal activity causing BOLD signal, the relationship between the neuronal activity and the hemodynamic processes on a larger scale remains an open question.
Since near-infrared spectroscopy (NIRS) ² is capable of capturing the individual characteristics of cerebral hemodynamics such as oxy-hemoglobin (HbO), deoxy-hemoglobin (HbR) and total hemoglobin (HbT) content, some researchers use NIRS to reveal the nature of the BOLD signal. Results showed great agreement between NIRS signals and BOLD, and stronger correlation between BOLD and HbR compared to HbO signals [179]. Flow response measure with ASL also had great correspondence with HbT signal. Investigation of connection between neuronal activity and NIRS captured characteristics [180] revealed the non-linear mapping between the neuronal activity and evoked hemodynamic processes. This result should be a red flag for those who try to define the general relation between neuronal activation and BOLD signal as mostly linear. The conjoint analysis of BOLD and NIRS signals revealed the silent BOLD signal during present neural activation registered by E/MEG modalities [157]. This mismatch between E/MEG and fMRI results is known as the se so y mo o a a o [181]. To
²NIRS is often used in the context of near-infrared optical imaging (NIOI), whenever multiph sensors allow to capture spatial distribution of the signal 84 explain this effect, the Vein and Capillary model was used to describe the BOLD signal in terms of hemodynamic parameters [157]. The suggested model permits the existence of silent and negative O responses during positive neuronal activation. The investigation of neurovascular coupling under influence of dopaminergic drugs showed that direct effects of dopamine upon the vasculature cannot be ignored
[182] and can provide further insights in the nature of negative O which can be due to interplay between vasoconstrictive and vasodilatory substances. These facts, together with an increasing number of studies [183] suggesting that sustained negative BOLD HR is a primary indicator of decreased neuronal activation, provide yet more evidence for the inherent complexity of BOLD HR. Recently published review [184] summarized more of known factors which shape the characteristics of BOLD signal at different levels: neuronal, vascular, and signal acquisition. This paper can be referred to as the summary on available at the moment knowledge on the nature of the BOLD signal. This section concludes by noting that the absence of a generative model of the BOLD response prevents the development of universal methods of multimodal analysis. Nevertheless, as discussed in this section and is shown by the results presented in the next section, there are specific ranges of applications where the linearity between BOLD and neuronal activation can be assumed. Such simplistic model can be voted for by the supported of Occam's razor principle which is to prefer simple models capable of describing the data of interest.
4.3 Existing Multimodal Analysis Methods
Whenever applicable, a simple comparative analysis of the results obtained from the conventional uni-modal analyses together with findings reported elsewhere, can be considered as the first confirmatory level of a multimodal analysis. This type of analysis is very flexible, as long as the researcher knows how to interpret the results and to draw 85 use u co c usio s es ecia y w e e e e esu s o com a iso e ea commo a i ies a i e e ces e wee e wo [1 7] O e o e a y e au a u imo a a a ysis makes imi e use o e a a om e mo a i ies a e cou ages esea c e s o ook o a a ysis me o s w ic wou i co o a e e a a ages o eac si g e mo a i y e e e ess sim e i s ec io is e u o awi g e imi a y co c usio s o e ausi i i y o e o m a y co oi a a ysis usi g o e o e me o s esc i e i is sec io i c u i g co e a i e a a ysis w ic mig e co si e e a i i ia a oac o y
4.3.1 Correlative Analysis of EEG and MEG with FMRI
I some e e ime s e E/MEG sig a ca se e as e e ec o o s o a eous eu o a ac i i y (e.g., e i e ic isc a ges o c a ges i e ocessi g s a es (e.g., igi a ce s a es e ime o se s e i e om E/MEG a e a o e a ua e o u e M I a a ysis w e e e BOLD sig a o e ca o o i e suc imi g i o ma io o i s a ce suc use o EEG a a is c a ac e is ic o e e e ime s e o me ia a Triggered fMRI acquisi io sc eme (Sec io 1 Co e a i e E/MEG/ M I a a ysis ecomes mo e i igui g i e e is a s o ge e ie i e i ea e e e cy e wee e BOLD es o se a ea u es o E/MEG sig a (e.g., am i u es o E eaks owe s o eque cy com o e s a e wee e emo y amics o e ai a e co es o i g a ame e o e esig (e.g., eque cy o s imu us ese a io o e e o s imu us eg a a io e E/MEG/ M I a a ysis e ec i e y e uces e i e e ias ese i e co e io a M I a a ysis me o s y emo i g e ossi e o - i ea i y e wee e esig a ame e a e e oke eu o a es o se e co e a i e a a ysis e ies o e e ocessi g o E/MEG a a o e ac e ea u es o i e es (E com o e s suc as 7 [1 5] o 3 [97] am i u e o 86 mismatch negativity (MMN) [100], contingent negative variation (CNV) [88], various characteristics of epileptic spikes [185]) to be compared with the fMRI time course with simple correlative or GLM approaches. The obtained E/MEG features first get convolved with a hypothetical HRF (Section 4.2.l) to accommodate for the HR sloppiness and are then subsampled to fit the temporal resolution of fMRI. The analysis of fMRI signal correlation with amplitudes of selected peaks of ERPs revealed sets of voxels which have a close to linear dependency between the BOLD response and amplitude of the selected ERP peak, thus providing a strong correlation (P < 0.001 [105]). A parametric experimental design with different noise levels introduced for the stimulus degradation [100, 105] or different levels of sound frequency deviant [100] helped to extend the range of detected ERP and fMRI activations, thus effectively increasing the significance of the results found. To support the suggested connection between the specific ERP peak and fMRI activated area, the correlation of the same BOLD signal with the other ERP peaks must be lower if any at all [105]. As a consequence, such analysis cannot prove that any specific peak of EEG is produced by the neurons located in the fMRI detected areas alone but it definitely shows that they are connected in the specific paradigm.
The search for the covariates between the BOLD signal and wide-spread neuronal signals, such as the alpha rhythm, remains a more difficult problem due to the ambiguity of the underlying process, since there are many possible generators of alpha rhythms corresponding to various functions [186] and was shown to have high variability in the rest state not only across subjects but also within a single subject [187]. As an example, Goldman et al. [92] and Laufs et al. [94] were looking for the dependency between fMRI signal and EEG alpha rhythm power during interleaved and simultaneous EEG/fMRI acquisition correspondingly. They report similar (negative correlation in parietal and frontal cortical activity), as well as contradictory (positive correlation) findings, which can be explained by the variations in the experimental setup [188] or by the heterogeneous coupling between the alpha rhythm and the BOLD response [94]. Consecutive study 7
[189] attempted to resolve alpha band response function (ARF) of the BOLD response. The ARF was found to differ significantly between different regions: mainly positive in the thalamus, similar in amplitude and time between occipital and left and right parietal areas, positive at the eyeball and negative at the back of the eye. It was suggested that ARF variation across different regions were due to regional differences in alpha band activity as well as due to the differences in haemodynamic response function of BOLD signal. Despite the obvious simplification of the correlative methods, they may still have a role to play in constraining and revealing the definitive forward model in multimodal applications.
3 p t n hn
The common drawback of the presented correlative analyses techniques is that they are based on the selection of the specific feature of the F/MEG signal to be correlated with the fMRI time trends, which are not so perfectly conditioned to be characterized primarily by the feature of interest. The variance of the background processes, which are present in the fMRI data and are possibly explained by the discarded information from the E/MEG data, can reduce the significance of the found correlation. That is why it was suggested [190] to use the entirety of the E/MEG signal, without focusing on its specific frequency band, to derive the E/MEG and fMRI signal components which have the strongest correlation among them. The introduction of decomposition techniques (such as basis pursuit, PCA, ICA, etc.) into the multimodal analysis makes this work particularly interesting. To perform the decomposition [190], Partial Least-Squares (PLS) regression was generalized into the tri-PLS2 model, which represents the F/MEG spectrum as a linear composition of trilinear components. Each component is the product of spatial (among E/MEG sensors), spectral and temporal factors, where the temporal factors have to be maximally correlated with the corresponding temporal component of the similar fMRI signal decomposition into bilinear components: products of the spatial and temporal factors. Analysis using tri-PLS2 modeling on the data from [92] found a decomposition into 3 components corresponding to alpha, theta and gamma bands of the EEG signal.
The fMRI components found had a strong correlation only in alpha band component
(Pearson correlation 0.83, p = 0.005), although the theta component also showed a linear correlation of 0.56, p = 0.070. It is interesting to note, that spectral profiles of the trilinear EEG atoms received with and without fMRI influence were almost identical, which can be explained either by the non-influential role of fMRI in tri-PLS2 decomposition of EEG, or just by a good agreement between the two. On the other hand, EEG definitely guided fMRI decomposition, so that the alpha rhythm spatial fMRI component agreed very well with the previous findings [92].
3 3 E v l nt C rr nt p l M d l
Equivalent Current Dipole (ECD) is the most elaborated and widely used technique for source localization in EMSI [see 4, for an overview of localization methods in E/MEG]. It can easily account for activation areas obtained from the fMRI analysis thus giving the necessary fine time-space resolution by minimizing the search space of non-linear optimization to the thresholded fMRI activation map. While being very attractive, such a method bears most of the problems of the ECD method (described in Chapter and introduces another possible bias due to the belief in the strong coupling between hemodynamic and electrophysiological activities. For this reason it needs to be approached with caution in order to carefully select the fMRI regions to be used in the ECD/fMRI combined analysis. Although good correspondence between ECD and fMRI results is often found [191], some studies reported a significant (1-5 cm) displacement between locations obtained from fMRI analysis and ECD modeling [89, 116, 192-194]. The displacement 9
often was found to be very consistent across the experiments of different researchers
• using the same paradigm (for instance motor activations [89, 90, 195]). To assess the importance of found displacements it is necessary to keep in mind proven lower bounds on localization errors adherent for the cases of precise head modeling and
mis-localization due to nuisance parameters (e g number and models of sources, tissue conductivities and their anisotropy, head geometry) misspecification which can
drastically degrade localization performance. As it was already mentioned, in the first step, a simple comparison of detected activations across the two modalities can be done to increase the reliability of dipole localization alone. Further, additional weighting by the distance from the ECD to the corresponding fMRI activation foci can guide ECD optimization [196] and silent in fMRI activations can be accommodated by introducing free dipoles without the constraint on dipole location. Auxiliary fMRI results can help to resolve the ambiguity of the inverse E/MEG problem if ECD lies in the neighborhood of multiple fMRI activations. Placing multiple ECDs inside the fMRI foci with successive optimization of ECDs orientations and magnitudes may produce more meaningful results, especially if it better describes the E/MEG signal by the suggested multiple ECDs model. Due the large number of consistent published fMRI results [197], it seems viable
to perform a pure E/MEG experiment with consequent ECD analysis using known
relevant fMRI activation areas found by the other researchers performing the same kind of experiment [198], thus providing the missing temporal explanation to the known fMRI
activations.
3 n r Inv r M th d
Dale and Sereno [199] formulated a simple but powerful linear framework for the integration of different imaging modalities into the inverse solution of DECD, where the 9
solution was presented as unregularized (just minimum-norm) (B.7) with W Q = Cs and
Wx C E .
The simplest way to account for fMRI data is to use thresholded fMRI activation map as the inverse solution space but this was rejected [200] due to its incapability to account for fMRI silent sources, which is why the idea to incorporate variance information from fMRI into Cs was further elaborated [201] by the introduction of relative weighting for fMRI activated voxels via constructing a diagonal matrix WQ = W M I = ii} where vii = 1 for fMRI activated voxels and v ii = 7/0 E [0, 1] for voxels which are not revealed by fMRI analysis. A Monte Carlo simulation showed that v = 0.1 (which corresponds to the 90% relative fMRI weighting) leads to a good compromise with the ability to find activation in the areas which are not found active by fMRI analysis and to detect active fMRI spots (even superficial) in the DECD inverse solution. An alternative formulation of the relative fMRI weighting in the DECD solution can be given using a subspace regularization (SSR) technique [202], in which an E/MEG source estimate is chosen from all possible solutions describing the E/MEG signal, and is such that it minimizes the distance to a subspace defined by the fMRI data (see Figure 4.2). Such formulation helps to understand the mechanism of fMRI influence on the inverse E/MEG solution: SSR biases underdetermined the E/MEG source locations toward the fMRI foci. The relative fMRI weighting was tested [203] in an MEG experiment and found conjoint fMRI/MEG analysis results similar to the results reported in previous fMRI, PET, MEG and intracranial EEG studies. Babiloni et al. [204] followed Dale et al. [203] in a high resolution EEG and fMRI study to incorporate non-thresholded fMRI activation maps with other factors. First of all, the W M I was reformulated to
(WfMRI)ii = v0 + (1 — v0) /Δma 9 where Δi corresponds to the relative change of the fMRI signal in the i-th voxel, and Δmax is the maximal detected change. This way the relative E/MEG/fMRI scheme is preserved and locations of stronger fMRI activations have higher prior variance. Finally the three available weighting factors were combined: 91 fMRI relative weighting, correlation structure obtained from fMRI described by the matrix of correlation coefficients Ks, and the gain normalization weighting matrix Wn :
WQ = W½fn½KsWfMRRIM IW ½½I Although W WfMRI alone had improved EMSI localization, the incorporation of the K s lead to finer localization of neuronal activation associated with finger movement.
Although most of the previously discussed DECD methods are involved in finding minimal norm solution, the fMRI conditioned solution with minimal L 1 norm
(regularization term in (B.5) C(Q) = II Q I I1) is shown to provide a sparser activation map [205] with activity focalized to the seeded hotspot locations [196]. An fMRI-conditioned linear inverse is an appealing method due to its simplicity, and rich background of DECD linear inverse methods derived for the analysis of E/MEG signals. Nonetheless, one should approach these methods with extreme caution in a domain where non-linear coupling between BOLD and neural activity is likely to overwhelm any linear approximation [193].
4.3.5 Beamforming
Lahaye et al. [206] suggest an iterative algorithm for conjoint analysis of EEG and fMRI data acquired simultaneously during an event-related experiment. Their method relies on iterated source localization by the linearly constrained minimum variance (LCMV) beamformer (B.9), which makes use of both EEG and fMRI data. The covariance Cx used by the beamformer is calculated anew each time step, using the previously estimated sources and current event responses from both modalities. This way neuronal sites with a good agreement between the BOLD response and EEG beamformer reconstructed source amplitude, benefit most at each iteration. Although the original formulation is cumbersome, this method appears promising as (a) it makes use of both spatial and temporal information available from both modalities, and (b) it can account 92
MEG only Wi M I A MEG Source Space C Subspace Defined by fMRI
B Minimum-Norm Estimate D Subspace Regularized Estimate
p SS
M
Figure 4.2 Geometrical interpretation of subspace regularization in the MEG/EEG source space. (A) The cerebral cortex is divided into source elements q 1 , q , , qK, each representing an ECD with a fixed orientation. All source distributions compose a vector q in K-dimensional space. (B) The source distribution q is divided into two components qa e Sa range(G ), determined by the sensitivity of MEG sensors and q° E null G, which does not produce an MEG signal. (C) The fMRI activations define E another subspace S '. (D) The subspace-regularized fMRI-guided solution qSS is closest to S '', minimizing the distance PqSSR 11 where P (a N x N diagonal matrix with P ii = 1/0 when the i-th fMRI voxel is active/inactive) is the projection matrix into the orthogonal complement of SiMi'. (Adapted from [202, Figure 1]) 93
o si e BOLD sou ces usi g a e ec o-me a o ic cou i g co s a w ic is es ima e o eac i o e a e i es e i ue ce o e BOLD sig a a a gi e oca io o o e es ima io o Cs w ic i u i es e es ima e o C •
43.6 Bayesian Inference
u i g e as eca e ayesia me o s ecame omi a i e o a i is ic sig a a a ysis e i ea e i em is o use ayes u e o e i e a posterior probability o a gi e hypothesis a i g o se e a a D, w ic se es as evidence o su o e y o esis
w e e p(7-1) a p(D) a e io o a i i ies o e y o esis a e i e ce
co es o i g y a e co i io a o a i i y p(DIH is k ow as a likelihood function. us ( 3 ca e iewe as a me o o com i e e esu s o co e io a ike i oo a a yses o mu i e y o eses i o e os e io o a i i y o e
y o eses p(H D) o some u c io o i a e ee e ose o e a a e e i e os e io o a i i y ca e use o se ec e mos o a e y o esis i.e., e o e wi e ig es o a i i y
ea i g o e ma imum a posteriori (MA es ima e w e e e io a a o a i i y p(D) (o e ca e a partition function) is omi e ecause e a a oes o e e o
e c oice o e y o esis a i oes o i ue ce e ma imi a io o e 7 -1 o e c ass o o ems e a e o e sig a ocessi g y o esis ge e a y co sis s o a mo e M c a ac e i e y a se o uisa ce a ame e s = } e ima y goa usua y is o i a MA es ima e o some qua i y o i e es A o mo e ge e a y i s os e io o a i i y is i u io (ΔI M Θ ca e a 9 arbitrary function of the hypothesis or its components A = 1 ( or often just a specific
nuisance parameter of the model A 7 - 1 To obtain posterior probability of the nuisance parameter, its marginal probability has to be computed by the integration over the rest of the parameters of the model
(4.5) Due to the integration operation involved in determination of any marginal probability, Bayesian analysis becomes very computationally intensive if analytical integral solution does not exist. Therefore, sampling techniques (e g MCMC, Gibbs sampler) are often used to estimate full posterior probability (ΔI .M), MAP
 = arg maxΔ (ΔI M o some s a is ics suc as a e ec e a ue E[ΔI .A4] of the quantity of interest.
The Bayesian approach sounds very appealing for the development of multimodal methods. It is inherently able to incorporate all available evidence, which is in our case obtained from the fMRI and E/MEG data ( = { } to support the hypothesis on the location of neuronal activations, which is in the case of DECD
model is 7-1 = {Q, A4}. However, the detailed analysis of (4.3) leads to necessary simplifications and assumptions of the prior probabilities in order to derive a computationally tractable formulation. Therefore it often loses its generality. Thus to derive a MAP estimator for ÔIX,B,M Trujillo-Barreto et al. [207] had to condition the computation by a set of simplifying modeling assumptions such as: noise is normally distributed, nuisance parameters of forward models have inverse Gamma prior distributions, and neuronal activation is described by a linear function of hemodynamic response. The results on simulated and experimental data from a somatosensory MEG/fMRI experiment confirmed the applicability of Bayesian formalism to the multimodal imaging even under the set of simplifying assumptions mentioned above. 95
Usually, model ,A4 is not explicitly mentioned in Bayesian formulations (such as
(4.5)) because only a single model is considered. For instance, Bayesian formulation of
LORETA E/MEG inverse corresponds to a DECD model, where e = Q is constrained to be smooth (in space), and to cover whole cortex surface. In the case of the Bayesian Model
Averaging (BMA), the analysis is carried out for different models M i , which might have different nuisance parameters, e.g., E/MEG and BOLD signals forward models, possible spatial locations of the activations, constraints to regularize E/MEG inverse solution. In BMA analysis we combine results obtained using all considered models to compute the posterior distribution of the quantity of interest
where the posterior probability p(MiI ID) of any given model M i is computed via B ayes' rule using prior probabilities p(M i ) , p(D) and the likelihood of the data given each model
Initially, BMA was introduced into the E/MEG imaging [208], where Bayesian interpretation of (B.7) was formulated to obtain p(QIX, F) for the case of Gaussian u co e a e oise (W = C = єI I o e o c ea e a mo e e ai o ume ge s a i io e i o a imi e se o s a ia y is i c u c io a com a me s w ic a e a i a i y com i e o e i e a M i , search space for the E/MEG inverse problem.
At the end, different models are sampled from the posterior probability p(M i lX) to get the estimate of the expected activity distribution of ECDs over all considered source models 9
where the normalized probability (Mi Bayes' Factor ic a io o s αi , are
I e o igi a MA amewo k o E/MEG [ ] α i = 1 Vi, i.e., the models had a flat prior PDF because no additional functional information was available at that point. Melie- García et al. [209] suggested to use the significance values of fMRI statistical t-maps to derive p(M) as the mean of all such significance probabilities across the present in
M i compartments. This strategy causes the models consisting of the compartments with significantly activated voxels get higher prior probabilities in BMA. The introduction of fMRI information as the prior to BMA analysis reduced the ambiguity of the inverse solution, thus leading to better localization performance. Although further analysis is necessary to define the applicability range of the BMA in E/MEG/fMRI fusion, it already looks promising because of the use of fMRI information as an additional evidence factor in E/MEG localization rather than a hard constraint. Due to the flexibility of Bayesian formalism, various Bayesian methods solving E/MEG inverse problem already can be easily extended to partially accommodate evidence obtained from the analysis of fMRI data. For instance, correlation among different areas obtained from fMRI data analysis can be used as a prior in the Bayesian reconstruction Of correlated sources [210]. The development of a neurophysiologic generative model of BOLD signal would allow many Bayesian inference methods (such as [211]) to introduce complete temporal and spatial fMRI information into the analysis of E/MEG data. 97
4.4 Suggested Multimodal Analysis Method
e o y easo some eo e ge os i oug is ecause i s u ami ia e i o y
— Paul Fix
4.4.1 Motivation
As it was shown in the previous sections, fMRI BOLD signal is an inherently non-linear function of neuronal activation. Nevertheless there have been multiple reports of linear dependency between the observed BOLD response and the selected set of E/MEG signal features. In general, such results are not inconsistent with the non-linearity of BOLD, since a non-linear function can be approximately linear in the context of a specific experimental design, regions of interest, or dynamic ranges of the selected features of E/MEG signals. Besides dominant LFP/BOLD linearity reported by Logothetis and Wandell [173] and also confirmed in the specific frequency bands of EEG signal during flashing checkerboard experiment [212], there have been reports of a strong correlation between the BOLD and the power of different frequency bands of E/MEG signals [213]. Besides conventionally explored frequency bands of E/MEG, very low-frequencies (f < 0.1 Hz, also known as DC-E/MEG) signal component has not yet been a subject of attention for multimodal integration despite recent experiments showing the strong correlation between the changes of the observed DC-EEG signal and the hemodynamic changes in the human brain [214]. In fact, such DC-E/MEG/BOLD coupling suggests that the integration of fMRI and DC-E/MEG might be a particularly useful way to study the nature of the time variations in HR signal. These variations are usually observed during fMRI experiments but are not explicitly explained by the experimental design or by the physics of MR acquisition process. 98
To summarize, independent researchers have detected various features of the E/MEG signals which have strong correspondence to the BOLD signal. Since the spectral power of E/MEG signal at different frequency bands was reported to correlate significantly with the BOLD response, it is worthwhile to approach the fusion problem by looking for a way to describe BOLD signal in terms of the time-varying spectral representation of E/MEG.
4.4.2 Formulation of the Approach
The overall goal of the presented analysis approach is to determine the mapping function .F, such that
where fMRI signal F at each voxel i is described in terms of the E/MEG data X via a transformation
Due to aforementioned loose or absent coupling between power of some frequency bands and the design of the experiment, it is desirable to analyze the data from both modalities, whenever they are acquired in a single session. Due to impossibility of acquiring *MEG in the magnet, suggested method should mostly be applicable to the conjoint analysis of EEG and fMRI. Suggested approach is similar to GLM analysis of fMRI data where each voxel is represented as a linear combination of a limited number of explanatory variables. However, it substantially differs from GLM analysis since it neither imposes a specific shape of HRF, nor, strictly speaking, it requires linear relationship between X and F i . Furthermore, it does not require the specification of the experimental design.
Richness of E/MEG signal (e.g., in terms of its spectral properties), absence of a generative forward model of BOLD signal, and sluggishness of BOLD response are the main obstacles toward deriving a reliable transformation function Slow evolution of 99
BOLD response requires to rely on a vast duration of E/MEG signal preceding any given volume of a BOLD signal. Therefore, to accommodate for the existing lag between neural activation and its reflection within fMRI signal, (4.8) can be further elaborated as
where t is a moment in time when an fMRI volume is acquired (fMRI volumes are usually evenly spaced in time with a fixed TR, typically 2-4 sec), and is a reasonable duration to look back in time for neural activations which should be significantly reflected in BOLD response at the given time point. Such setup is schematically depicted in Figure 4.3. In most of the scenarios, it is reasonable to take = 1 + TR sec. The duration of 10 sec should be sufficient to account for the typical HRF delay. The duration of TR is included since different slices of a volume are acquired at different offsets within the TR. Hence for atypically long TRs (as it is in the experimental data of Section 4.6, where TR=10 sec) it is necessary to consider sufficient amount of E/MEG data to describe any possible voxel in the acquired volume.
It is possible to account for the offset of any given slice while formulating (4.9), if slice ordering is known in advance. But such procedure might be inappropriate since fMRI data are conventionally motion corrected as a part of preprocessing. Motion correction
relies on the interpolation between different slices, hence it can smear the effect of slice acquisition timing. Furthermore, transformation .F could be described as a composition of actual regression function g and some (optional) time-frequency transformation (e.g., SFFT, Wavelet standard or packet decomposition)
where does not depend on the voxel in question and is introduced to convert the original E/MEG data into the feature space which is known to have correspondence with BOLD response. Transformation can also incorporate multiple characteristics of the 1
igu e 3 Mapping of neural data from EEG into fMRI. During learning, for each fMRI voxel a single mapper is trained on a corresponding time-window of EEG data to predict fMRI signal in that voxel. During testing the trained mapper predicts fMRI signal. Comparison between predicted (in red) and real time course of fMRI (in green) allows to assess the performance of the mapper. 101
same feature of the E/MEG signal, . ., for a single time-frequency slot it can provide both amplitude and the power of the E/MEG signal. That can allow to account for some non-linear effects which were reported for the BOLD response without requiring transformation g itself to be non-linear. Such approach is substantially different from the existing multimodal methods. None of the existing methods (such as correlative analysis Section 4.3.1) presented a reliable and generative (defining the actual transformation .F) way to describe BOLD signal in terms of multiple components of E/MEG signal. Furthermore, they often had to rely on some chosen, hence rigid, HRF function. That heavily restricts the amount of novel conclusions to be drawn from the data analysis.
Having selected features of the signals to be involved in the fusion, many EMSI methods could naturally be extended to account for fMRI data if a generative forward model of BOLD signal was available. For instance, direct universal-approximator inverse methods [215, 216] have been found to be very effective (fast, robust to noise and to
complex forward models) for the E/MEG dipole localization problem, and could be augmented to accept fMRI data if the generative model for it was provided. Due to the mentioned forward model of BOLD and abundance of the features of E/MEG signal, instead of relying on a overly too flexible generic architecture for the description of the
signal ( . ., artificial neural networks as in [215, 216]), it is logical to use some other
supervised machine learning regression approaches ( . ., SVR and GPR) which are inherently well regularized, and as a result are capable to provide reasonable generalization performance even if the dimensionality of input space greatly exceeds the number of available data samples.
4.4.3 Promises
As it was shown in Chapter 3, methods such as GPR and SVM are capable of near-perfect generalization performance on classification tasks, even when 1 dimensionality of input space greatly exceeded number of data samples. Furthermore, supervised learning methods encourage model testing to provide unbiased estimates of the generalization performance for a derived transformation At last, many methods
( . ., linear ones) conveniently provide feature-weighting, which can be used for determining the components of the signal relevant for a given classification or regression task. As it will be shown in the followup sections, derivation of a reliable transformation J can help to
• identify brain regions, where fMRI signal is relatively well explained by the information present in E/MEG signal. It provides insights not only about the nature
of the fMRI response, but also about interconnectivity among the areas, since deep structures themselves are weakly represented in the E/MEG signal, hence goodness of their description in terms of the E/MEG signal relies on their connections with more exterior parts of the brain;
• localize the regions which are related to specific components of E/MEG signal ( . ., different frequency bands);
• provide sensitivity maps across the regions for the E/MEG per each channel, which in turn would allow to characterize coherence among distant parts of the brain due to inherent connectivity;
• filter E/MEG signal based on fMRI data, or even to perform electrode rejection based on its participation in the description of fMRI data;
• filter, interpolate, or predict (thus increase temporal sampling) of fMRI signal based on the E/MEG signal;
• assess spatial evolution of HRF of BOLD fMRI response per each voxel; 103
4.4.4 Computational Efficiency
Since the suggested method proceeds in a mass-univariate fashion by estimating transformation function g per each target voxel, it sounds like an idea destined for a failure since typical fMRI volume has up to 200,000 brain voxels (actual number of voxels depends on the spatial resolution of the data). However, in the suggested approach each input sample for the regression is the set of features of E/MEG data, which are constant across all voxels. The time-course of fMRI voxel is the only varying variable, different per each voxel. Therefore, most of the kernel based ML methods, for a specific set of parameters, need to compute the kernel matrices only once. Initial optimization point for the iterative ML methods (such as constrained quadratic optimization in SVM) can be pre-seeded with previously found solution for another voxel. Due to guaranteed uniqueness of the solution, such initialization of optimization would result in faster convergence, and must result in nearly exact (up to a specified numerical tolerance) solution as if it was done with full training and arbitrary initial starting point for
optimization. That allows to perform training of the regression per each voxel in a
reasonable amount of time (e.g., from ten to hundreds of milliseconds), therefore making whole brain analysis feasible. Due to the mass-univariate nature, estimation can also be parallelized for the computation in the high performance computing environments.
For the analysis presented here, SVR was chosen due to its simple parametrization
(just the trade-off coefficient C with a reasonable default) and computational efficiency. Comparison of the results obtained with different regression methods is left for future research.
4.5 Validation on a Simulated Data
As previously emphasized, any novel methodology has to be validated first on data with the known characteristics of the noise and of the signal of interest. Due to the absence of a 1
ea is ic s u y o e a om i is ecessa y o simu a e e sig a a oise co i io s is c a e esc i es e o oco use o simu a e e a ase a ese s e esu s o e a a ysis usi g e sugges e me o o ogy
5 1 S l t n
Simu a e e i o me co sis e o a si g e ac i e s ice wi i a ee s e ( ai sku sca s e ica mo e o e ea (see igu e M I eso u io was ake o e iso o ic 3 mm A is eso u io e mo e e " ai " co sis e o 1 o e s em o a eso u io s o e simu a e sig a s we e ake o e i i e wi e o es o ea a ase (ki y o i e y e ma [ 17] a a ysis o w ic wi e ese e i Sec io 5 o EEG a (equi a e o =5 sec o M I ee egio s o i e es ( OIs co o e i e a k e a ue i igu e we e e i e o ca y e e e a e (E ac i i y w e eas a o e oca io s ca ie o y s o a eous ac i i y amou o w ic e e e o e y e o e ma e (g ay o w i e o EEG mo e i g eac o e co ai e a si g e i o e o ie a io o w ic ( e a ows i igu e was se a o g e o ma o e a i icia o i g ( e ic e i ye ow o e co ica issue Eig ou o 1 EEG se so s ( a e e as SO S7 we e oca e i e a e o M I s ice O e 7 (S S 1 we e oca e i e s ice a -way o e o o e ea a e as c a e S 15 was oca e o o o e ea o wa mo e o
EEG sig a was es ima e usi g O e MEEG³ (see igu e C 3 wo y es o eu a ac i a io s we e mo e e — e e - e a e (E a s u ious ac i i y E e - e a e ac i i y was a e o y wi i e e i e OIs O se s o E ac i i y we e ake om e e e ime a esig i e ea a ase o Sec io e a i e am i u es i e OI O13 (see igu e o e ac i a io s we e i e a ge om 5 o 3 e e i g o e owe o au i o y s imu a io ( s
³ http://www-sop.inria.fr/odyssee/software/OpenMEEG 105
igu e Simulated Environment Space. 2148 voxels in a single plane. 3 ROIs were defined to carry ER activity. Arrows (in red) visualize the orientation of dipoles within gray matter voxels. 8 out of 16 EEG sensors are located on the scalp surface. and the laterality of the area voxel belonged to in respect to the side of auditory stimulation (left vs right).
Since various parts of the brain are active all the time, there always exists neural activity which is not related to the activity caused by the external stimulation, or the experimental design in general. Nevertheless, such spurious activity effects both EEG and M I responses but does not correlate with the design, thus it cannot be accounted for in G M analysis of M I or in the E analysis of EEG. Hence, this activity contributes variance to the acquired neural data which cannot be explained by the variables of the experimental design, and it is different from the instrumental noise which should not have common cause in different data modalities. For the simulation, a range of different types of spurious activity were assigned randomly to different locations (see Figure C.1 for examples of the shapes of E responses). Amplitude and frequency of occurrence of spurious activity varied depending on the type of the simulated matter of the brain. Within 1
(a) Event-related (b) Total (ER and spurious) (c) Acquired
Figure 4.5 ERP of the simulated EEG data. gray matter voxels spurious activity occurred with a frequency of 2 events per second and relative amplitude of 0.6. Within white matter voxels frequency and amplitude were half a small.
While modeling EEG responses, all simulated neural activity was jitterred uniformly within 50 msec of stimuli appearance to simulate the variability of the response per trial. Additionally, all locations in the "brain" were assigned an additional jitter (once again random within 50 msec) to model the variability of the response due to variability of the activation depending on the location. Both types of activations contributed to the total local field potential (LFP) signal assigned to each voxel. LFP signals of all of the brain voxels were transformed into EEG signals through the aforementioned forward EEG model. Jitterring and addition of spontaneous activity resulted in ERP responses, which looked very similar to the ones acquired empirically
(see Figure 4.5(b)). Additional additive source of variance, instrumental noise, was modeled by random sampling of normal distribution with mean 0 and variance I with a consecutive low-pass Butterworth filtering in temporal domain with stop-band frequency of 4 Hz (see Figure 4.6).
Simple linear convolutional model (see Section 4.2.1) of fMRI signal was chosen.
Squared values of the activations amplitudes were convolved with a conventional HRF
(Figure C.2). Additive instrumental noise was modeled in the fashion similar to EEG 1 7
r Spectral characteristics of simulated EEG signals. The upper part presents sample durations of EEG. The lower part shows corresponding spectrograms. noise, but with a different stop-band frequency of the filter at 0.2 Hz. Samples of the modeled signals are presented on the Figure 4.7.
5 t r p r t n
EEG recordings of brain activity are dominated by the periods of rhythmic activity. It was shown [218] that characteristic bands of EEG signal, as generated by the neural matter, are centered at frequencies arranged according to the powers of = 1.61803399 (the "go e
a io " ) (see Figure 4.8).
Thus, prior the analysis, EEG signal of each channel was transformed into its power spectra, and was summarized by the total power in each temporal/spectral bin taking = 10 sec (see (4.10)) seconds prior the onset of each fMRI volume acquisition. Temporal width of each bin was taken to be 500 ms and frequencies followed aforementioned 0 distance in a log space to cover the range up to 25 Hz available within simulated signal sampled at 50 Hz. That resulted in
4http://en.wikipedia.org/wiki/Golden_ratio 1
igu e 7 Samples of the simulated data from all data modalities at the chosen location, which carries event-related activity. EEG signal is represented by a sensor S 15 which is located on the top of the "head". Event-related plots (in purple) depict simulated signal which is caused solely by ER activity. Total signal (in green) shows total signal caused by all neural activity, which includes both ER and spontaneoús components. Acquired signal (in black) shows total signal with additive noise, which should mimic the real data obtained empirically. 1 9
igu e Mu i e mo a eak eque cies o e sis e y ms ge e a e i iso a e eoco e i i o (A a e om [ 1 igu e 1] 11
( ime- i s 9(s ec a - i s 1 (e ec o es = ea u es i EEG a ase w ic a e same um e o sam es as e um e o simu a e M I o umes Eac ea u e o EEG a eac o e i M I we e i e e e y s a a i e ( - sco e io e a a ysis A s a a - o c oss- a i a io oce u e was u e eac simu a e o e usi g i ea S (Su o ec o є-i se si i e eg essio o a oisy a a (EEG a M I o o ai es ima es o ow we is eg essio cou e ic u see M I a a
5 3 S n l n tr t n
eco s uc io e o ma ce o ime-se ies was c a ac e i e y co e a io coe icie e wee a ge ( o o se e u i g ai i g a e ic e a a Goo ess o e ic io o oisy u see a a ac oss a o e s o e ai is ese e o igu e 9 Image o e e s ows o y o e s w ic asse es o i g a =3 e a i e o y-c a ce e o ma ce is s ou esu o y i 1% a se osi i es i e esu s (i.e., o a e age o e s om e o u a io a a s o assess is i u io o y-c a ce e o ma ces es o i g e ie o e assum io a e is og am o e o ma ces is a mi u e o wo is i u io s o ma is i u io ce e e a w ic co es o s o y-c a ce eco s uc io e o ma ce a e o e a i a i y s a e is i u io o a i eco s uc io e o ma ces w ic oes o a e a y sig i ica um e o ega i e a ues Suc assum io ea i y a ows o assess e s a a e ia io o e is i u io o y-c a ce e o ma ce sim y y symme i i g ega i e e o ma ce a ues a ou a com u i g e s a a e ia io o e o ai e is i u io Si ce is a a ysis is o e o e simu a e a a i is mo e a ua e o u ge e e o ma ce o e me o y i s ec i g e eco s uc io e a i e o e oise ess a a igu e 1 s ows a e eco s uc io e o ma ce is sig i ica y ig e w e 111
igu e 9 es o e co e a io coe icie s e wee oisy a e ic e M I a a 112
Figure 4.10 Thresholded correlation coefficients between clean and predicted fMRI data. compared to the noiseless data, and that the distribution of by-chance performances is nicely segregated from the distribution of the meaningful reconstructions.
To visually inspect the goodness of reconstruction, time series for four chosen representative voxels are presented on Figure 4.11. These voxels correspond to a single voxel per each ROI (randomly chosen within ROIs) and a voxel at midbrain. Activity of the midbrain voxel should not possibly be reflected in EEG signal, thus neither reliably reconstructed. Figure 4.1 1 shows that time series within ROI1 and RO12 were relatively well reconstructed, especially in respect to the clean original signal, although the learner was exposed during training only to the noisy EEG and fMRI data. 113
igu e 11 Original and predicted fMRI time series for representative voxels. CC stands for correlation coefficient (or Pearson's correlation) with a corresponding significance level RMSE/RMS_t is the ratio between root-mean squared error (RMSE) and root- mean square of the target signal (RMS_t). Perfect reconstruction would correspond to CC= 1.0 and RMSE/RMS _t=0.0. 11
igu e 1 Sensitivities of SVR trained to predict fMRI signal of a voxel in OI 1 Each plot corresponds to a single channel of EEG and represents sensitivities in time (horizontal) and frequency (vertical) bins.
5 Se si i i y A a ysis
Demonstrated reconstruction of the original time-series within the ROI voxels allows to state, that the regression has learned information about neural activation which was embedded in both EEG and fMRI signals. It allows further to inspect the sensitivities of the learned regression. Figure 1 shows regression's sensitivity for the representative voxel within O11 which is neighboring SO and S7 electrodes of EEG. Sensitivities show that results are highly sensitive to the high frequencies components of the SO. 115
Figure 4.13 Aggregate sensitivity of SVR to the high-frequencies of SO EEG electrode while predicting a voxel in ROI 1. Error-bars correspond to the standard error estimate across 4 splits of data.
Aggregate sensitivity (sum of powers of the sensitivities) in the range of high- frequencies for that sensor is presented in Figure 4.13. Maximum sensitivity nicely matches the time-lag of 5 sec where HRF used for fMRI simulation reaches its maximum value (see Figure C.2). Absence of the sensitivity to the lower frequencies of EEG signal could be attributed to the fact that simulated instrumental noise added to EEG signal was low-pass filtered, hence contributing mostly to the lower frequencies, thus making them less reliable for the estimation of the BOLD response. Obtained results allow to conclude that the analysis of the regression sensitivities provides means to assess characteristics of the underlying HRF of the BOLD response. 11
Appl t n t l EEG/ M I t
e a a ysis o e simu a e a a ese e i e e ious sec io as a i a e e ia i i y o e sugges e me o is sec io ese s e esu s o a yi g e ac y e same a a ysis wo k ow o ea a a a a om a si g e su ec o is a a ysis was ki y o i e y e ma e ai s a ou a a acquisi io se u e ocessi g s ages a e esu s o co e io a a a ysis usi g G M a E me o s cou e ou i e o igi a u ica io [ 17]
1 t r p r t n
EEG a M I a a we e acqui e simu a eous y i a si g e sessio w i e a su ec was e o mi g a sim e au i o y ask a i e e e e s o s imu a io M I a a co sis e o 1 7 o umes o s ices eac ( ick ess mm w ic we e acqui e a e i - a e eso u io o 3 mm ue o is-sy c o i a io o M I a EEG acquisi io equi me s omi a M I o 1 9 sec was a us e o e 1 731 sec o ma c e ime c ock o EEG acquisi io a wa e Suc was e uce y e a a ysis EEG a a w ic ca ies ea y a i ac s w ic occu w e e e M I acquisi io is i og ess (see Sec io 1 1 M I a a was e ocesse usi g S oo s s a ia y smoo e a W M o mm a em o a y ig - ass i e e a e cu o co es o i g o 1 sec O y ai o e s we e se ec e o e a a ysis EEG a a was co ec e o emo e M -a i ac s (see [ 17] o e e ai s em o a y ow- ass i e e a cu -o eque cy o a ow sam e a sam i g eque cy o 5 o ma c em o a eso u io o e simu a e a a A EEG a a was e- e e e ce o e mea e wee 9 a 1 e ec o es (see igu e C o e e ec o es aceme i a y ica EEG sys em 1 ou o 9 EEG se so s (C C C 7 9 1 7 we e se ec e o e uce e i u ime sio a i y o 117
e a a ese se so s we e se ec e as e ese a i e o o i i g c ea E es o se o e s imu i
S n l n tr t n
A a ysis wo k ow was e same as o e simu a e a a i Sec io 5 3 wi a si g e i e e ce I is a ase was e a i e y ig (1 sec w ic mea s a ac ua acquisi io o e s ices was s ea oug a u a io o o e y o e seque ce o s ice acquisi io = sec was ake o i eg a e amou o i o ma io su icie o accou i g o M I a a i a ou s ices eco s uc io e o ma ce o e a a was s a is ica y sig i ica ac oss a a ie y o ai egio s ( igu e 1 ime se ies o ou c ose o e s om i e e a s o e ai a e s ow i igu e 1 esu s a e su isi g i e se se a e eco s uc io ac ie e e a i e y ig e o ma ce o o y i e e e io a eas w ic a e oca e i ici i y o EEG e ec o es u a so i me ia a eas w ic esuma y s ou e weak y e ese e i EEG sig a s egis e e o e sca is e ec ca e a i u e o e ese s o g ac i a io i ose a eas w i e e o mi g a gi e ask I ee a e a eas e ec e wi G M a a ysis ( igu e 15 a e a so ese o e es o e eco s uc io ma ( igu e 1 A i io a y e au i o y co e w ic was o sig i ica y ac i e acco i g o e G M esu s was eco s uc e a a com a a e e e wi e ig au i o y co e A ossi e e a a io cou e a mo u a e ac i a io i e e au i o y co e so a G M a a ysis w ic aims a e ec i g co sis e ac i a io ac oss e ia s is o a e o accou o suc a ia ce e ce i is i ca a e o e ec e ac i a io 11
igu e 1 Thresholded correlation coefficients between real and predicted fMRI data in a single subject from [217].
igu e 15 GLM analysis: significant (thresholded at IzI < 2.0) activation (in red) and deactivation (in blue) in response to auditory stimulation. Slices are plotted in radiological convention, hence left side of the brain is presented to the right from inter-hemispheric point. 119
Figure 4.16 Original and predicted fMRI time series for representative voxels. 1
igu e 17 Sensitivities of SVR trained to predict fMRI signal of a voxel with right auditory cortex. Each plot corresponds to a single channel of EEG and represents sensitivities in time (horizontal) and frequency (vertical) bins.
3 Se si i i y A a ysis
Figure 4.17 shows the sensitivity estimates of the SVR for a single voxel. Clearly there is higher amount of variance in sensitivities compared the sensitivities obtained on the simulated data (Figure 4.12). Nevertheless, sensitivities to the signal components at lower frequencies seems to be consistent with the assumption of the delayed BOLD response.
Aggregate sensitivity for low frequencies of the sensor Cz is presented in Figure 4.18.
Since for this analysis, EEG signal of duration covering acquisition of two fMRI volumes
(20 sec) was used as input to the regression, it seems to be logical that fMRI signal at a given time point could be sensitive to the activity which is reflected in the previous or in the next volume (depending on the slice position).
Many studies (see Sections 4.3.l and 4.3.2) aim to discover the brain regions which are generators of specific rhythms of cortical activity. Analysis of the SVR sensitivities to different spectral bands could allow to detect the areas of the brain where a particular frequency band contributes the most variance to the reconstruction of fMRI BOLD signal. 121
Figure 4.18 Aggregate sensitivity of SVR to the low-frequencies of Cz EEG electrode while predicting a voxel in the right auditory cortex. 122
igu e 19 Agg ega e se si i i y o e eque cies i e α- a a ge o eque cies
Figure 4.19 shows thresholded (z > 2.0 under assumption of normally distributed values) aggregate sensitivities to the EEG signal spectral components in the a -band. Highest sensitivities are found in the occipital areas. Such finding agrees with previous studies
[92, 187]. Unfortunately, limited number of the slices in the available fMRI data does not cover the full brain, therefore makes it impossible to explore all the brain regions in regards to the localization of generators of different frequency bands. Thresholded sensitivities for other frequency bands from this analysis can be found in the Appendix C
(Figures C.5, C.6, and C.7).
Another possible dimension for the sensitivities aggregation is the sensitivity plots of the voxels per each EEG sensor. Figure 4.20 shows thresholded aggregated sensitivity of different voxels to the signal of T8 EEG sensor, which was positioned in the neighborhood of the temporal cortex (therefore index "T" in the name). Results show that information from T8 sensor primarily targets description of the areas such as 1 3 au i o y co e w ic a e oca e i se so s ici i y e e e ess some o e is a a eas a e a so se si i e o e a a i o i s a ce a ie a egio s w ic acco i g o G M esu s (see igu e 15 a e o e e ime - e a e e e e ess ca y O sig a w ic is we esc i e y EEG (see igu e 1 W e e ose egio s a e us weak y e ec e i c a e o ac ua y ac i a e co e e y wi au i o y co e emai s a o e ques io Si ce i is u ike y o e ec o e o ick u sig i ica amou o a ia ce om is a a eas o e co a- a e a emis e e o e cou a so s ecu a e a su a- es o e o e s co a- a e a o e ci e o a e e e y
oi s o e co ica s uc u e om e co us co osum5 w ic aci i a es e commu ica io e wee wo emis e es e ce suc sim e a a ysis o e co s uc e ma e se si i i ies ca ossi y a ow o e uce u c io a co ec i i y e wee is a a eas Agg ega e se si i i ies o o e e ec o es a so seems o e i acco a ce wi ei s acia oca io s o e am e sig i ica se si i i ies o C 1 a C e ec o es co e a ie a a eas ( igu es C C 9 C 1 i A e i C
7 C n l n
is c a e ese e a o e a oac owa e a a ysis o mu i e a a mo a i ies Sugges e a oac i o im ose a y s ic o wa mo e o M I es o se w ic makes is mo e a o ia e o e e is i g me o s esu s o e a a ysis o simu a e a ea a a s owe ossi i i y o c ea e a e ia e ma i g o EEG a a i o e s ace o M I sig a s Suc ma i g a owe o i e i y e egio s o e ai w ic we e mos ac i e u i g e e e ime u we e o e ec e y e co e io a a a ysis o M I a a u e mo e a a ysis o e co s uc e ma i g a owe o i e i y e egio s e a e o e s eci ic y ms o EEG sig a as we as o isco e ossi e u c io a co ec io s om e a eas i ici i y o EEG se so s o some is a a eas
//e wiki e ia o g/wiki/Co us_co osum5 124
Figure 4.20 Aggregate sensitivity to the signal of T8 EEG channel (see Figure C.4 for the typical location of the sensor in respect to the brain). CHAPTER 5
PYMVPA: MACHINE LEARNING FRAMEWORK FOR NEUROIMAGING
ece y ou e Machine learning open source software ¹ o ec s ows a im essi e o e e ess s i i com e e sam e o a ai a e so wa e ackages w ic im eme M me o s A e e y eas s a i g wi ese a ea y a ai a e ig -qua i y so wa e i a ies as e o e ia o acce e a e scie i ic og ess i e eme gi g ie o c assi ie - ase a a ysis o ai -imagi g a a A oug ese i a ies a e ee y a ai a e ei usage y ica y assumes a ig - e e o og ammi g e e ise a s a is ica o ma ema ica k ow e ge e e o e i wou e o g ea a ue o a e a u i yi g amewo k a e s o i ge we -es a is e eu oimagi g oo s a mac i e ea i g so wa e ackages a o i es ease o og amma i i y c oss i a y i eg a io a a s a e M I a a a i g Suc amewo k s ou a eas a e e i e o owi g ea u es
User-centered programmability with an intuitive user interface Si ce mos eu oimagi g esea c e s a e o a so com u e scie is s i s ou equi e o y a mi ima amou o og ammi g a i i y Wo k ows o y ica a a yses s ou e su o e y a ig - e e i e ace a is ocuse o e e e ime a esig a a guage o e eu oimagi g scie is a ei g sai o cou se a i e aces s ou a ow access o e ai e i o ma io a ou e i e a ocessi g o com e e si e e e si i i y i a y easo a e ocume a io is a ima y equi eme
Extensibility I s ou e easy o a su o o a i io a e e a mac i e ea i g oo o es o e e u ica i g e e o a is ecessa y w e a si g e a go i m as o e im eme e mu i e imes
¹ http://www.mloss.org
125 126
Transparent reading and writing of datasets Because the toolbox is focused on neuroimaging data, the default access to data, should require little or no specification for the user. The toolbox framework should also take care of proper conversions into any target data format required for the external machine learning algorithms.
Portability It should not impose restrictions about hardware platforms and should be able to run on all major operating systems.
Open source software It should be open source software, as it allows one to access and to investigate every detail of an implementation, which improves the reproducibility of experimental results, leading to more efficient debugging and gives rise to accelerated scientific progress [16].
PyMVPA, a Python-based toolbox for multivariate pattern analysis of neuroimaging data, was designed to meet all the above criteria for a classifier-based analysis framework. Following sections provide only a rough overview about the most important aspects of the toolbox. However, a comprehensive user manual is available [29] to provide more details and typical code examples.
5.1 Design
One of the main goals of PyMVPA is to reduce the gap between the neuroscience and ML communities. To reach this goal, PyMVPA was designed to provide a convenient, easy to
use, community developed (free and open source ² ), and extensible framework to facilitate use of ML techniques on neural information. PyMVPA combines Python data processing, visualization, and basic I/O facilities together with I/O code and examples tailored for
² PyMVPA is distributed under an MIT license, which complies with both Eree Software and Open Source definitions. The software is freely available in source and in binary form from the project website http://www.pymvpa.org 1 7
eu oscie ce a e is s a um e o y o mo u es w ic mig e o i e es i e eu oscie i ic co e o a easy s a i o yM A a M I e am e a ase [a si g e su ec om e s u y y 5] is a ai a e o ow oa om e yM A we si e As a e ig ig e yM A is o e o y M amewo k a ai a e o sc i i g a i e ac i e a a e o a io i y o I co as o some o e ima i y GUI- ase M oo o es (e.g., O a ge E e a yM A is esig e o o i e o us a oo o u a amewo k o co cise ye i ui i e sc i i g o ossi y com e a a ysis i e i es o ac ie e is goa yM A o i es a um e o ui i g ocks a ca e com i e i a e y e i e way igu e i Sec io s owe a sc ema ic e ese a io o e amewo k esig i s ui i g ocks a ow ey ca e com i e i o com e e a a ysis i e i es yM A co sis s o se e a com o e s (g ay o es suc as M a go i ms o a ase s o age aci i ies Eac com o e co ai s o e o mo e mo u es (w i e o es o i i g a ce ai u c io a i y e.g., classifiers, u a so ea u e-wise measu es [e.g., I- E IE ; 3 ] a ea u e se ec io me o s [ ecu si e ea u e e imi a io E; 35 3 ] y ica y a im eme a io s wi i a mo u e a e accessi e oug a u i o m i e ace a ca e e o e e use i e c a gea y i.e., a y a go i m usi g a c assi ie ca e use wi a y a ai a e c assi ie im eme a io suc as support vector machine [S M; 17] o sparse multinomial logistic regression [SM ; 37] Some M mo u es o i e ge e ic meta a go i ms a ca e com i e wi e basic im eme a io s o M a go i ms o e am e a Multi-Class me a c assi ie o i es su o o mu i-c ass o ems e e i a u e yi g c assi ie is o y ca a e o ea wi i a y o ems A i io a y yM A makes use o a um e o e e a so wa e ackages ( ack o es i igu e i c u i g o e y o mo u es a ow- e e i a ies a com u i g e i o me s (e.g., R³ ). I e case o SVM, c assi ie s a e i e ace o e
³ http://www.r-project.org 1 implementations in Shogun or I S M PyMVPA only provides a convenience wrapper to expose them through a uniform interface. Using externally developed software instead of reimplementing algorithms has the advantage of a larger developer and user base and makes it more likely to find and fix bugs in a software package to ensure a high level of quality. However, using external software also carries the risk of breaking functionality when any of the external dependencies break. To address this problem PyMVPA utilizes an automatic testing framework performing various types of tests ranging from unittests
(currently covering 84% of all lines of code) to sample code snippet tests in the manual and the source code documentation itself to more evolved "real-life" examples. This facility allows one to test the framework within a variety of specific settings, such as the unique combination of program and library versions found on a particular user machine. At the same time, the testing framework also significantly eases the inclusion of code by a novel contributor by catching errors that would potentially break the project's functionality. Being open-source does not always mean easy to contribute due to various factors such as a complicated application programming interface (API) coupled with undocumented source code and unpredictable outcomes from any code modifications
(bug fixes, optimizations, improvements). PyMVPA welcomes contributions, and thus, addresses all the previously mentioned points:
A b l t f r d nd d nt t n All the source code (including website and examples) together with the full development history is publicly available via a distributed version control system, which makes it very easy to track the development of the project, as well as to develop independently and to submit back into the project. 129
Inplace code documentation Large parts of the source code are well documented using
reStructuredText4 , a lightweight markup language that is highly readable in source format as well as being suitable for automatic conversion into HTML or PDF
reference documentation. In fact, Ohloh.net5 source code analysis judges
PyMVPA as having "extremely well-commented source code".
Developer guidelines A brief summary defines a set of coding conventions to facilitate uniform code and documentation look and feel. Automatic checking of compliance to a subset of the coding standards is provided through a custom
PyLint6 configuration, allowing early stage minor bug catching.
Moreover, PyMVPA does not raise barriers by being limited to specific platforms. It could fully or partially be used on any platform supported by Python (depending on the availability of external dependencies). However, to improve the accessibility, convenient ways are provided for the installation across a variety of platforms: binary installers for Windows, and MacOS X, as well as binary packages for Debian GNU/Linux (included in the official repository), Ubuntu, and a large number of RPM-based. GNU/Linux distributions, such as OpenSUSE, RedHat, CentOS, Mandriva, and Fedora. Additionally, the available documentation provides detailed instructions on how to build the packages from source on many platforms.
By providing simple, yet flexible interfaces, PyMVPA is specifically designed to connect to and use externally developed software. Any analysis built from those basic elements can be cross-validated by running them on multiple dataset splits that can be generated with a variety of data resampling procedures [e.g., bootstrapping, 38]. Detailed information about analysis results can be queried from any building block and can be visualized with various plotting functions that are part of PyMVPA, or can be mapped
4http://en.wikipedia.org/wiki/ReStructuredText http://www.ohloh.net/projects/pymvpdfactoids 6http://www.logilab.org/projects/pylint 130 back into the original data space and format to be further processed by specialized tools
(i.e., to create an overlay volume analogous to a statistical parametric mapping). The solid arrows in Figure 2.l represent a typical connection pattern between the modules. Dashed arrows refer to additional compatible interfaces which, although potentially useful, are not necessarily used in a standard processing chain. A final important feature of PyMVPA is that it allows, by design, researchers to compress complex analyses into a small amount of code. This makes it possible to complement publications with the source code actually used to perform the analysis as supplementary material. Making this critical piece of information publicly available allows for in-depth reviews of the applied methods on a level well beyond what is possible with verbal descriptions. To demonstrate this feature, this paper is accompanied by the full source code to perform all analyses shown in the following sections.
5.2 Dataset Handling
Input, output, and conversion of datasets are a key task for PyMVPA. A dataset representation has to be simple enough to allow for maximum interoperability with other toolkits, but simultaneously also has to be comprehensive in order to make available as much information as possible to e.g., domain-specific analysis algorithms. One of the key features of PyMVPA is its ability to read fMRI datasets and transform them into a generic format that makes it easy for other data processing toolboxes to inherit them. PyMVPA comes with a specialized dataset type for handling import from and export to images in the NIfTI or inferior ANALYZE formats. It automatically configures an appropriate mapper by reading all necessary information from the NIfTI file header. Upon export, all header information is preserved (including 131 em e e a s o ma io ma ices is makes i e y easy o o u e ocessi g i a y o e I I-awa e so wa e ackage ike A I ai oyage 8 S 9 o SPM¹°. Si ce ma y a go i ms a e a ie o y o a su se o o e s yM A o i es co e ie me o s o se ec o e s ase o OI masks Successi e y a ie ea u e se ec io s wi e ake i o accou y e ma i g a go i m o I I a ase s a e e se ma i gs om e ew su s ace o ea u es i o e o igi a a as ace e g o isua i a io is au oma ica y e o me u o eques owe e e ma e co s uc i yM A w ic is a ie o eac a a sam e is mo e e i e a a sim e 3- a a o ume o - ea u e ec o a s o ma io e o igi a a as ace is o imi e o ee ime sio s o e am e w e a a y i g a e e ime usi g a e e - e a e a a igm i mig e i icu o se ec a si g e o ume a is e ese a i e o some e e A ossi e so u io is o se ec a o umes co e i g a e e i ime w ic esu s i a ou - ime sio a a as ace A ma e ca a so e easi y use o EEG/MEG a a e g ma i g s ec a ecom osi io s o e ime se ies om mu i e e ec o es i o a si g e ea u e ec o yM A o i es co e ie me o s o ese use-cases a a so su o s e e se ma i g o esu s i o e o igi a a as ace w ic ca e o a y ime sio a i y
5.3 Machine Learning Algorithms
yM A i se oes o a ese im eme a ossi e c assi ie s e e i a we e esi a e Cu e y i c u e a e im eme a io s o a k- ea es - eig o c assi ie as we as i ge e a i e ogis ic ayesia i ea Gaussia ocess (G a s a se mu i omia ogis ic eg essio s [SM ; 37] owe e i s ea o is i u i g ye a o e im eme a io o o u a c assi ica io a go i ms e oo o e i es a ge e ic c assi ie
7http://afni.nimh.nih.gov/afni 8http://www.brainvoyager.com 9http://www.fmrib.ox.ac.uk/fsl ¹°http://www.fil.ion.ucl.ac.uk/spm 13 i e ace a makes i ossi e o easi y c ea e so wa e w a e s o e is i g mac i e ea i g i a ies a e a e ei c assi ie s o e use wi i e yM A amewo k A e ime o is w i i g w a e s o su o ec o mac i e a go i ms [S M; 17] o e wi e y use LIBSVM ackage [ 19] a Shogun mac i e ea i g oo o [ ] a e i c u e A i io a c assi ie s im eme e i e s a is ica og ammi g a guage a e o i e wi i yM A [e.g., eas a g e eg essio A S 1] e so wa e w a e s e ose as muc u c io a i y o e u e yi g im eme a io as ecessa y o a ow o a seam ess i eg a io o e c assi ica io a go i m i o yM A W a e c assi ie s ca e ea e a e a e e ac y as a y o e a i e im eme a io s Some c assi ie s a e s eci ic equi eme s a ou e a ase s ey ca e ai e o o e am e su o ec o mac i es [17] o o o i e a i e su o o mu i-c ass o ems i.e., isc imi a io o mo e a wo c asses o ea wi is ac yM A o i es a amewo k o c ea e me a-c assi ie s (see igu e 1 ese a e c assi ie s a u i i e se e a asic c assi ie s o ose im eme e i yM A a ose om e e a esou ces a a e eac ai e se a a e y a ei es ec i e e ic io s a e use o o m a oi me a- e ic io some imes e e e o as boosting [see ] esi es ge e ic mu i-c ass su o yM A o i es a um e o a i io a me a-c assi ie s e.g., a c assi ie a au oma ica y a ies a cus omi a e ea u e se ec io oce u e io o ai i g a e ic io A o e e am e is a me a-c assi ie a a ies a a i a y ma i g a go i m o e a a o im eme ime sio a i y a s o ma io a /o e uc io s e suc as i ci a com o e a a ysis ( CA i e e e com o e a a ysis (ICA o usi g im eme a io s
om MDP ¹¹ o wa e e ecom osi io ia pywavelets¹² es i e i s ig - e e i e ace yM A o e s e ai e i o ma io o e use o ac ie e a use u e e o a s a e cy a c assi ie s ca easi y s o e a y amou o
¹¹ http://mdp-toolkit.sourceforge.net ¹²http://www.pybytes.com/pywavelets/ 133 additional information. For example, a logistic regression might optionally store the output values of the regression that are used to make a prediction. PyMVPA provides a framework to store and pass this information to the user if it is requested. The type and size of such information is in no way limited. However, if the use of additional computational or storage resources is not required, then it can be switched off at any time, to allow for an optimal tradeoff between transparency and performance. This section presents only a brief overview of PyMVPA organization and coding approaches. Significantly broader and in-depth coverage of PyMVPA is available from the "PyMVPA User Manual" [29]. The main goal of the section is to demonstrate the flexibility of PyMVPA for the construction of various analysis pipelines; and also to highlight the conciseness of the actual code needed to perform a desired statistical learning analysis. That theoretically should allow shipment of complete analysis code with any publication as supplemental material (e.g., see [23]).
5.4 Example Analyses
This section will demonstrate the functionality of PyMVPA by running some selected analyses on fMRI data from a single participant (participant l) from a dataset published by Haxby et al. [25]. This dataset was chosen for demonstration examples because, since its first publication, it has been repeatedly reanalyzed [13, 26, 28], was analyzed on the course of this dissertation work (see Section 3.4.2 for the description of the experiment and data preprocessing), and is readily available from PyMVPA's website. For the sake of simplicity, analysis was focused on the binary CATS vs. SCISSORS classification problem. These two categories were chosen for the demonstration of PyMVPA facilities since their classification in general poses harder problem, than of more "specialized" ones, such as FACE and HOUSE (analysis on which was presented in the Section 3.4.3). All volumes recorded during either CATS or 134
SCISSORS blocks were extracted and voxel-wise Z-scored with respect to the mean and standard deviation of volumes recorded during rest periods. Z-scoring was performed individually for each run to prevent any kind of information transfer across runs. Every analysis is accompanied by source code snippets that show their implementation using the PyMVPA toolbox. For demonstration purposes they are limited to the most important steps.
5.4.1 Loading a Dataset
Dataset representation in PyMVPA builds on NumPy arrays. Anything that can be converted into such an array can also be used as a dataset source for PyMVPA. Possible formats range from various plain text formats to binary files. However, the most important input format from the functional imaging perspective is NIfTI ¹³ , which PyMVPA supports with a specialized module. The following short source code snippet demonstrates how a dataset can be loaded from a NIfTI image. PyMVPA supports reading the sample attributes from a simple two- column text file that contains a line with a label and a chunk id for each volume in the NIfTI image (line 0). To load the data samples from a NIfTI file it is sufficient to create a i ft i a ase object with the filename as an argument (line 1). The previously-loaded sample attributes are passed to their respective arguments as well (lines 2-3). Optionally, a mask image can be specified (line 4) to easily select a subset of voxels from each volume based on the non-zero elements of the mask volume. This would typically be a mask image indicating brain and non-brain voxels.
¹³ o a ce ai eg ee yM A a so su o s im o i g A A Y E i es 135
Once the dataset is loaded, successive analysis steps, such as feature selection and classification, only involve passing the dataset object to different processing objects. All following examples assume that a dataset was already loaded.
5.4.2 Simple Full-brain Analysis
The first analysis example shows the few steps necessary to run a simple cross-validated classification analysis. After a dataset is loaded, it is sufficient to decide which classifier and type of splitting shall be used for the cross-validation procedure. Everything else is automatically handled by C oss a i a e a s e E o The following code snippet performs the desired classification analysis via leave-one-out cross-validation. Error calculation during cross-validation is conveniently performed by
a s e E o which is configured to use a linear C-SVM classifier ¹ on line 6. The leave-one-out cross-validation type is specified on line 7.
Simply passing the dataset to cv (line 8) yields the mean error. The computed error defaults to the fraction of incorrect classifications, but an alternative error function can be passed as an argument to the a s e E o call. If desired, more detailed information is available, such as a confusion matrix based on all the classifier predictions during cross-validation.
¹ LIBSVM C-SVC [219] with trade-off parameter C being a reciprocal of the squared mean of Frobenius norms of the data samples. 136
5.4.3 Multivariate Searchlight
One method to localize functional information in the brain is to perform a classification analysis in a certain ROI [e g 223]. The rationale for size, shape and location of a ROI can be e g anatomical landmarks or functional properties determined by a GLM-contrast. Alternatively, Kriegeskorte et al. [20] proposed an algorithm that scans the whole brain by running multiple ROI analyses. The so-called mu i a ia e sea c ig runs a classifier-based analysis on spherical ROIs of a given radius centered around any voxel covering brain matter. Running a searchlight analysis computing e g generalization performances yields a map showing where in the brain a relevant signal can be identified while still harnessing the power of multivariate techniques [for application examples see 19, 224].
A searchlight performs well if the target signal is available within a relatively small area. By increasing size of the searchlight the information localization becomes less specific because, due to the anatomical structure of the brain, each spherical ROI will contain a growing mixture of gray-matter, white-matter, and non-brain voxels. Additionally, a searchlight operating on volumetric data will integrate information across brain-areas that are not directly connected to each other i e located on opposite borders of a sulcus. This problem can be addressed by running a searchlight on data that has been transformed into a surface representation. PyMVPA supports analyses with spatial
searchlights (not extending in time), operating on both volumetric and surface data (given an appropriate mapping algorithm and using circular patches instead of spheres). The searchlight implementation can compute an arbitrary measure within each sphere. In the following example, the measure to be computed by the searchlight is configured first. Similar to the previous example it is a cross-validated transfer or generalization error, but this time it will be computed on an odd-even split of the dataset and with a linear C-SVM classifier (lines 9-11). On. line 12 the searchlight is setup to compute this measure in all possible 5 mm-radius spheres when called with a dataset 137
igu e 5 1 Searchlight analysis results for CATS vs. SCISSORS classification for sphere radii of 1, 5, 10 and 20 mm (corresponding to approximately 1, 15, 80 and 675 voxels per sphere respectively). The upper part shows generalization error maps for each radius. All error maps are thresholded arbitrarily at 0.35 (chance level: 0.5) and are not smoothed to reflect the true functional resolution. The center of the best performing sphere (i.e., lowest generalization error) in right temporal fusiform cortex or right lateral occipital cortex is marked by the cross-hair on each coronal slice. The dashed circle around the center shows the size of the respective sphere (for radius 1 mm the sphere only contains a single voxel).
MNI-space coordinates (x , y, z) in mm for the four sphere centers are: 1 mm (RI): (48, -61, -6), 5 mm (R5): (48, -69, -4), 10 mm (R 0): (28, -59, -12) and 20 mm (R20): (40, -54, -8). The lower part shows the generalization errors for spheres centered around these four coordinates, plus the location of the univariately best performing voxel (L1: -35, -43, -23; left occipito-temporal fusiform cortex) for all radii. The error bars show the standard error of the mean across cross-validation folds.
(line 13) The final call on line 14 transforms the computed error map back into the original data space and stores it as a compressed NIfTI file. Such a file can then be viewed and further processed with any NIfTI-aware toolkit.
9 CV = C oss a i a e a s e E o ( o transfer_e o = a s e E o Mi ea CS MC splitte =O E e S i e 1 Si = Sea c ig ( c radius =5) 13 sl_map = .s1(dataset) 1 dataset. map2Nifti ( si_map ). save( ' searchlight_5mm . nii . gz ') 13
igu e 5 1 s ows e sea c ig e o ma s o e CA S s SCISSO S c assi ica io o si g e o umes om e e am e a ase o a ii o 1 5 1 a mm es ec i e y U i i i g o y a si g e o e i eac s e e ( mm a ius yie s a ge e a i a io e o as ow as 17% i e es e o mi g s e e w ic is oca e i e e occi i o- em o a usi o m co e Wi i c eases i e a ius e e is a e e cy o u e e o e uc io i ica i g a e c assi ie e o ma ce e e i s om i eg a i g sig a om mu i e o e s owe e e e c assi ica io accu acy is ac ie e a e cos o e uce s a ia ecisio o sig a oca i a io e is a ce e wee e ce e s o e es - e o mi g s e es o 5 a mm sea c ig s o a s a mos 1 mm e owes o e a e o i e ig occi i o- em o a co e wi % is ac ie e y a sea c ig wi a a ius o 1 mm e es e o mi g s e e wi mm a ius (1 % ge e a i a io e o is ce e e e wee ig i e io em o a a usi o m gy us I com ises a o ima e y 7 o e s a e e s om ig i gua gy us o e ig i e io em o a gy us a so i c u i g a s o e ce e e um a e a e a e ic e I e e o e i c u es a sig i ica o o io o o e s sam i g ce e os i a ui o w i e ma e i ica i g a a s e e o is si e is o o ima gi e e s uc u a o ga i a io o e ai su ace K iegesko e e a [ ] sugges a a s e e a ius o mm yie s ea -o ima e o ma ce owe e w i e is assum io mig e a i o e ese a io s o o ec o e ies o ow- e e isua ea u es a sea c ig o is si e cou miss sig a s e a e o ig - e e cog i i e ocesses a i o e se e a s a ia y is i c u c io a su sys ems o e ai
5 t r S l t n
ea u e se ec io is a commo e ocessi g s e a is a so ou i e y e o me as a o a co e io a M I a a a a ysis i.e., e i i ia emo a o o - ai o e s is
asic u c io a i y is o i e y i i a ase as i was s ow o i e o o i e 139 an initial operable feature set. Likewise, a searchlight analysis also involves multiple feature selection steps (i.e., ROI analyses), based on the spatial configuration of features. Nevertheless, PyMVPA provides additional means to perform feature selection, which are not specific to the fMRI domain, in a transparent and unified way. Machine learning algorithms often benefit from the removal of noisy and irrelevant features [see 35, Section V.l. "The features selected matter more than the classifier used"]. Retaining only features relevant for classification improves learning and generalization of the classifier by reducing the possibility of overfitting the data. Therefore, providing a simple interface to feature selection is critical to gain superior generalization performance and get better insights about the relevance of a subset of features with respect to a given contrast. Table 5.l shows the prediction error of a variety of classifiers on the full example dataset with and without any prior feature selection. For classifiers with feature selection the classifier algorithm is followed by the sensitivity measure the feature selection was based on (e.g., LinSVM on 50(SVM) reads: linear SVM classifier using 50 features selected by their magnitude of weight from a trained linear SVM). Most of the classifiers perform near chance performance without prior feature selection I5 , and even simple feature selection (e.g., some percentage of the population with highest scores on some measure) boosts generalization performance significantly of all classifiers, including the non-linear algorithms radial basis function
SVM and, kNN. PyMVPA provides an easy way to perform feature selections. The ea u eSe ec io C assi ie is a meta-classifier that enhances any classifier with an arbitrary initial feature selection step. This approach is very flexible as the resulting classifier can be used as any other classifier, e.g., for unbiased generalization
¹5C a ce e o ma ce wi ou ea u e se ec io was o e o m o a ca ego y ai s i e a ase Eo e am e e S M c assi ie ge e a i e we o o e ai s o ca ego ies (e.g., ACE s OUSE wi ou io ea u e se ec io Co seque y SCISSO S s CA S was c ose o o i e a mo e i icu a a ysis case 1
es i g usi g C oss a i a e a s e E o o i s a ce e o owi g e am e s ows a c assi ie a o e a es o y o 5% o e o e s a a e e ig es A O A sco e ac oss e a a ca ego ies i a a icu a a ase I is o ewo y a e sou ce co e ooks a mos i e ica o e e am e gi e o i es 5- wi us e ea u e se ec io me o a e o i o c a ges a e ecessa y o e ac ua c oss- a i a io oce u e
I is im o a o em asi e a ea u e se ec io ( i es 17-19 i is case is not e o me i s o e u a ase w ic cou ias ge e a i a io es ima io I s ea ea u e se ec io is ei g e o me as a a o c assi ie ai i g us o y e ac ua ai i g a ase is isi e o e ea u e se ec io ue o e u i ie i e ace i is ossi e o c ea e a mo e so is ica e e am e w e e ea u e se ec io is e o me ia recursive 1 1
On line 25 the main classifier is defined, which is reused in many aspects of the processing: line 28 specifies that classifier to be used to make the final prediction operating only on the selected features, line 31 instructs the sensitivity analyzer to use it to provide sensitivity estimates of the features at each step of recursive feature elimination, and on line 33 we specify that the error used to select the best feature set is a generalization error of that same classifier. Utilization of the same classifier for both the sensitivity analysis and for the transfer error computation prevents us from re-training a classifier twice for the same dataset. The fact that the RFE approach is classifier-driven requires us to provide the classifier with two datasets: one to train a classifier and assess its features sensitivities and the other one to determine stopping point of feature elimination based on the transfer error. Therefore, the ea u eSe ec io C assi ie (line 27) is wrapped within a S i C assi ie (line 26), which in turn uses o S i e (line 39) to generate a set of data splits on which to train and test each independent classifier. Within each data split, the classifier selects its features independently using RFE by computing a generalization error estimate (line 33) on the internal validation dataset generated by the splitter. Finally, the S i C assi ie uses a customizable voting strategy (by default MaximalVote) to derive the joint classification decision. As before, the resultant classifier can now simply be used within
C oss a i a e a s e E o to obtain an unbiased generalization estimate of the trained classifier. The step of validation onto independent validation dataset is often overlooked by the researchers performing RFE [35]. That leads to biased generalization estimates, since otherwise internal feature selection of the classifier is driven by the full dataset. Fortunately, some machine learning algorithms provide internal theoretical upper bound on the generalization performance, thus they could be used as a
a s e _e o criterion (line 33) with RFE, which eliminates the necessity of 1 a i io a s i i g o e a ase Some o e c assi ie s e o m ea u e se ec io i e a y (e g SM a so see igu e 5 w ic emo es e u e o e e a e ici ea u e se ec io a a i io a a a s i i g
5 5 C n l n
ume ous s u ies a e i us a e e owe o c assi ie - ase a a yses a esse wi s a is ica ea i g eo y o e ac i o ma io a ou e u c io a o e ies o e ai e ious y oug o e e ow e sig a - o- oise a io [ o e iews see 1 5] A oug e s u ies co e a oa a ge o o ics om uma memo y o isua e ce io i is im o a o o e a ey we e e o me y e a i e y ew esea c g ou s is may e ue o e ac a e y ew so wa e ackages a s eci ica y a ess c assi ie - ase a a yses o M I a a a e a ai a e o a oa au ie ce Suc ackages equi e a sig i ica amou o so wa e e e o me s a i g om asic o ems suc as ow o im o a ocess M I a ase s o mo e com e o ems suc as e im eme a io o c assi ie a go i ms is esu s i a i i ia o e ea equi i g sig i ica esou ces e o e ac ua eu oimagi g a ase s ca e a a y e e yM A oo o aims o e a so i ase o co uc i g c assi ie - ase a a yses I co as o e 3 s m ugi o A I [31] i o ows a mo e ge e a a oac y o i i g a co ec io o commo a go i ms a ocessi g s e s a ca e com i e wi g ea e i i i y Co seque y e i i ia o e ea o s a a a a ysis o ce e M I a ase is acqui e is sig i ica y e uce ecause e oo o a so o i es a ecessa y im o e o a e ocessi g u c io a i y yM A is s eci ica y u e owa s e a a ysis o eu a a a u e ge e ic esig o e amewo k a ows o wo k i o e omai s equa y we e e i e a ase a i g a ows o e o easi y e e i o o e a a o ma s w i e a e same ime e e i g e ma i g a go i ms o e ese o e a a s aces a me ics 143
owe e e key ea u e o yM A is a i o i es a u i o m i e ace o i ge om s a a eu oimagi g oo s o mac i e ea i g so wa e ackages is i e ace makes i easy o e e e oo o o wo k wi a oa a ge o existing so wa e ackages w ic s ou sig i ica y e uce e ee o eco e a ai a e a go i ms o e co e o ai -imagi g esea c Mo eo e a e e a a i e a c assi ie s ca e ee y com i e wi e c assi ie -i e e e a go i ms o e.g., ea u e se ec io maki g is oo o a i ea e i o me o com a e i e e c assi ica io a go i ms e i o uc io o is c a e as is e o a i i y as o e o e goa s o a o ima a a ysis amewo k yM A co e is es e o e o a e ac oss mu i e a o ms a i s imi i g se o esse ia e e a e e e cies i u as o e o e o a e I ac yM A o y e e s o a mo e a e y ece e sio o y o a e um y ackage A oug yM A ca make use o o e e e a so wa e e u c io a i y o i e y em is com e e y o io a A oug yM A aims o e es ecia y use - ie y i oes o o i e a g a ica use i e ace (GUI e easo o o i c u e suc a i e ace is a e oo o e ici y aims o e cou age o e com i a io s o a go i ms a e e e o me o ew a a ysis s a egies a a e o easi y o eseea e y a GUI esig e e oo o is e e e ess use - ie y e a i g esea c e s o co uc ig y com e a a yses wi us a ew i es o easi y ea a e co e I ac ie es is y aki g away e u e o ea i g wi ow- e e i a ies a o i i g a g ea a ie y o a go i ms i a co cise amewo k e equi e ski s o a o e ia yM A use a e o muc i e e om eu oscie is s usi g e asic ui i g ocks ee e o o e o e es a is e M I a a ysis oo ki s (e.g., s e sc i i g o A I a S comma i e oo s o Ma a -sc i i g o S M u c io s
¹ Nothing prevents a software developer from adding a GUI to the toolbox using one of the many GUI toolkits that interface with Python code, such as PyQT (http://www.riverbankcomputing.co . uk/software/pyqt) or wxPython (http://www.wxpython.org/). 144
Recent releases of PyMVPA added support for visualization of analysis results, such as, classifier confusions, distance matrices, topography plots, and time-locked signals plots. However, while PyMVPA does not provide extensive plotting support it nevertheless makes it easy to use existing tools for MRI-specific data visualization. Similar to the data import PyMVPA's mappers make it also trivial to export data into the original data space and format, e g using a reverse-mapped sensitivity volume as a statistical overlay, in the same fashion to SPM in conventional analysis. This way PyMVPA can fully benefit from the functionality provided by the numerous available MRI toolkits.
The features of PyMVPA outlined here cover only a fraction of the currently implemented functionality. More information is, however, available on the PyMVPA project website, which contains user manual [29] with an introduction into the main concepts and the design of the framework, a wide range of examples, a comprehensive module reference as a user-oriented summary of the available functionality, and finally a more technical reference for extending the framework.
The emerging field of classifier-based analysis of fMRI data is beginning to complement the established analysis techniques and has great potential for novel insights into the functional architecture of the brain. However, there are a lot of open questions how the wealth of algorithms developed by those motivated by statistical learning theory can be optimally applied to brain-imaging data (e g fMRI-aware feature selection
algorithms or statistical inference of classifier performances). The lack of a go s a a for classifier-based analysis demands software that allows one to apply a broad range of available techniques and test an even broader range of hypotheses. PyMVPA tries to reach this goal by providing a unifying framework that allows to easily combine a large number of basic building blocks in a flexible manner to help neuroscientists to do rapid initial data exploration and, consecutive custom data analysis. PyMVPA even facilitates the integration of additional algorithms in its framework that are not yet discovered by 145
Figure 5.2 Feature selection stability maps for the CATS vs. SCISSORS classification. The color maps show the fraction of cross-validation folds in which each particular voxel is selected by various feature selection methods used by the classifiers listed in Table 5.l: 5% highest ANOVA F-scores (A), 5% highest weights from trained SVM (B), RFE using SVM weights as selection criterion (C), internal feature selection performed by the SMLR classifier with penalty term 0.1 (D), 1 (E) and 10 (F). All methods reliably identify a cluster of voxels in the left fusiform cortex, centered around MNI: -34, -43, -20 mm. All stability maps are thresholded arbitrarily at 0.5 (6 out of 12 cross-validation folds). 146 neuro-imaging researchers. Despite being able to perform complex analyses, PyMVPA provides a straightforward programming interface based on an intuitive scripting language. The availability of more user-friendly tools, like PyMVPA, will hopefully attract more researchers to conduct classifier-based analyses and, thus, explore the full potential of statistical learning theory based techniques for brain-imaging research. 1 7
bl 5 1 e o ma ce o a ious C assi ie s wi a wi ou ea u e Se ec io
ai i g a s e C assi ie ea u es E o ime E o ime u i i e (sec (+s e (sec Without feature selection i S M(C= e 91 5 7 + 7 i S M(C=1 e 91 5 7 3 + 7 i S M(C=1 91 5 3 + 7 S M( 91 5 3 5 + 7 1 I ( 91 5 + 3 9 With feature selection SM (1m= 1 31 19 9+ 3 1 SM ( m=1 9 5 11+ 3 1 SM ( m=1 9+ 3 1 S M o SM (1m=1 o - 5 11+ k o 5%(A O A 1 5 + 5 k o 5 (A O A 5 1 7+ k o SM (1m=1 o - 5 1 + i S M o 5%(S M 1 5 3 1 1 + 1 i S M o 5 (S M 5 7 3± i S M o 5%(A O A 1 5 1 13+ 1 i S M o 5 (A O A 5 9+ 3 i S M o SM (1m=1 o - 9 5 3 + i S M o SM (1m=1 o - 5 1 + 3 i S M+ E( - o 5 7 1 1 + 3 3 i S M+ E(O E e 9 9 + CHAPTER 6
SUMMARY
e e is o suc i g as ai u e o y esu s wi some mo e success u a o e s
— e Ke e A i u e is E e y i g I c
6.1 Conclusion
is isse a io ese e a u i ie me o o ogy owa e a a ysis o eu a a a om i e e o e e mu i e eu a a a mo a i ies e emo s a e a oac is ge e ic e oug o e use ac oss i e e e e s o i es iga io o ai u c io w i e e o ci g e ia e eco i g e o ma ce e a oac e su es u iase es i g o esu s a a ows co c usio s o e aw a ou eu a ocesses o i e es u e mo e a a ysis o co s uc e eco e s o i es e asis o a essi g i e i ica io a e assessme o s eci ici y i a way o ossi e wi co e io a a oac es e esu s o ai e wi is a oac e ea e e com o e s o e sig a s w ic wou o e e ec e usi g co e io a a oac es eco i g o e ace u a eco i gs a owe o a i a e ese ce o s imu i-s eci ic i o ma io a eco i g ci e as we as o assig s imu us s eci ic a e s o e i ci a u c io a u i s (i e eu o s I es iga io o EEG a a eco e se si i i ies a owe o e ec a e s imu us-se si i e com o e o e E A a ysis o o ec s eci ic isua ocessi g a owe o su o e y o esis o is i u e eu a co es a e a se ce o e A u ique y i o e i uma ace ecog i io u e mo e ma i g o e eu a a a om o e imagi g mo a i y i o a o e a owe e seg ega io o eu a a a com o e s ese wi i e em i ica a a a
148 149
o i e e mea s o e o mi g a a ie y o a i io a a a ysis a a ge om a a i e i g o s a ia oca i a io o e eu a ocesses
6.2 Contributions Beyond The Dissertation
e sig a eco i g amewo k ese e i is isse a io is a ica e o a um e o su - ie s i eu oscie ce om e ace u a eco i gs o c oss-mo a a a a a ysis e a ue o is a oac as ee emo s a e wi a um e o ese a io s a eu oscie ce co e e ces a se e a a ic es i ee - e iewe ou a s [ 3 ] is amewo k as a so s imu a e a a ie y o co a o a i e e o s ac oss a ious as ec s o uma ai u c io suc as causa i e e ce [ ] a a ge sca e eco i g [ 7] o aci i a e e a ica io o e a a ysis a oac ese e e e a y o - ase oo o yM A as ee e e o e i a oi e o wi Mic ae a ke (Ge ma y e Se e e g (USA a Ema ue e O i e i (I a y [ 3] As a so wa e o uc yM A as sig i ica y ma u e o e e as ew yea s a as a ea y ee use y a um e o esea c g ou s a i i i ua esea c e s a ou e g o e i c u i g ose wo ki g a MI UC A a a um e o o e u i e si ies a esea c i s i u es yM A as ee e e o e as a o E sy (Experimental
Psychology') o ec wi i e Debian GNU/Linux² o e a i g sys em o w ic I am³ o e o e ou e e o e s e E sy o ec aims o o i e e es ee a o e -sou ce esea c e i o me o syc o ogis s a eu oscie is s a ious so wa e o uc s (e.g., yE S y I I O e MEEG etc.) ge ma e o esea c o e ai we e ei e e e o e o ackage a mai ai e wi i E sy ue o e e i ic ackagi g sys em o e ia E sy ma e ese so wa e o uc s ea i y a ai a e usi g a si g e c ick o sim e comma i e o e a io ese ackages a e i http://alioth.debian.org/projects/pkg-exppsy/ ²http://www.debian.org ³ http://qa.debian.org/[email protected] 15 available not only to the thousands of Debian users, but to the users of Debian-derived distributions (e.g., Ubuntu) on a variety of platforms and architectures.
3 t r W r
The results that were presented here have demonstrated the power of statistical learning methods applied toward the analysis of neural data. Nevertheless, there are a number of ways in which the suggested framework can be improved and extended.
Although the results that have been presented here were based on constructing a linear relation between input and output data, the proposed framework is in no way limited to linear mapping. Non-linear mapping is of particular value for cross-modal data mapping which was presented in the Chapter 4. The main obstacle in adopting non- linear methods is the necessity to explore the larger parameter space of more flexible non- linear models. It is computationally demanding and often requires ad-hoc optimization. Although some ability to do model selection is already present in PyMVPA, the capacity for transparent model selection within PyMVPA is not yet implemented. Development of such a feature would rapidly facilitate the exploration of non-linear learners in a variety of applications.
At the moment, cross-modal mapping presented in the Chapter 4 makes no use of the EEG forward model. However, although the incorporation of a known forward model might significantly boost reconstruction performance, it is not yet implemented. Although multivariate methods, and decoders in particular, make use of the covariance and causality structure embedded in the data, the results presented here do not actually reveal any such effective connectivity. Nevertheless, preliminary results (not yet published) demonstrate the viability of using statistical learning regressions and associated sensitivity estimates to discover functional connectivity among different brain regions. Further development of such methods would be another goal for future research. 151
A oug e o ose amewo k o e s co si e a e a ue o esea c e s s u yi g e ai eco e s a s a is ica ea i g o o o e a a acea o a esea c o ems i eu oscie ce e success o is me o o ogy equi es a esea c e s co si e we o e esea c ques io a e e e ime a esig A y a icu a i es iga io o ai u c io w ic mig seem o ious a i s sig ee s o e a oac e wi co si e a e cau io a omai k ow e ge U o u a e y i ec a ica io s o c assi ie s [ ] o syc o ogica y com e o ems suc as ece io e ec io [ 9] ca e o e y naive o e ac ica [ 3 ] us a o e goa o e u sue is e e e o me o a gui e o esea c e s a wi aci i a e e o mu a io o esea c ques io s a e e ime a esig a ma imi es e a ue o is amewo k APPENDIX A
FORWARD MODELING OF EEG AND MEG SIGNALS
The analysis of E/MEG signals often relies on the solution of two related problems. The
o wa o em concerns the calculation of scalp potentials (EEG) or magnetic fields near the scalp (MEG) given the neuronal currents in the brain, whereas the i e se o em involves estimating neuronal currents from the observed E/MEG data. The difficulty of solving the forward problem is reflected in the diversity of approaches that have been tried (see [231] for an overview and unified analysis of different methods). The basic question posed by both the inverse and forward problems is how to model any neuronal activation so that the source of the electromagnetic field can be mapped onto the observed E/MEG signal. Assuming that localized and synchronized primary currents are the generators of the observed E/MEG signals, the most successful approach is to model the i-th source with a simple Equivalent Current Dipole (ECD)
[232], uniquely defined by three factors: location represented by the vector r i , strength qi , and orientation coefficients O The orientation coefficient is defined by projections of the vector qi into o ogo a Ca esia a es θi = qi /qi. However, the orientation coefficient may be expressed by projections in two axes in the case of a MEG spherical model where the silent radial to the skull component has been removed, or even, just in a single axis if normality to the cortical surface is assumed. The ECD model made it possible to derive a tractable physical model linking neuronal activation and observed
E/MEG signals. In case of K simultaneously active sources at time t the observed E/MEG
signal at the sensor xi positioned at pi can be modeled as
152 153 where G is a lead field function which relates the i-th dipole and the potential (EEG) or magnetic field (MEG) observed at the j-th sensor; and € is the sensor noise. In the given formulation, function G( i( pi ) returns a vector, where each element corresponds to the lead coefficient at the location p i generated by a unit-strength dipole at position r i (t) with the same orientation as the corresponding projection axis of θi . The inner-product between the returned vector and dipole strength projections on the same coordinate axes yields a j-th sensor the measurement generated by the i-th dipole. The forward model (A.1) can be solved at substantial computational expense using available numerical methods [233] in combination with realistic structural information obtained from the MRI data. This high computational cost is acceptable when the forward model has to be computed once per subject and for a fixed number of dipole locations, but it can be prohibitive for dipole fitting, which requires a recomputation of the forward model for each step of non-linear optimization. For this reason, rough approximations of the head geometry and structure are often used: e.g., best-fit single sphere model which has a direct analytical solution [234] or the multiple spheres model to accommodate for the difference in conductivity parameters across different tissues. Recently proposed MEG forward modeling methods for realistic isotropic volume conductors [235, 236] are more accurate and faster than Boundary Elements Method (BEM), and hence may be useful substitutes for both crude analytical methods and computationally intensive finite-element numeric approximations. APPENDIX B
EEG AND MEG INVERSE PROBLEM
B.1 Equivalent Current Dipole Models
e E/MEG i e se o em is e y c a e gi g (see [ 37 3 ] o a o e iew o me o s i s i e ies o e so u io o e o wa o em w ic ca e com u a io a y e e si e es ecia y i e case o ea is ic ea mo e i g Seco e
ea - ie u c io g om (A is o - i ea i i so a e o wa mo e e e s o - i ea y o e oca io s o ac i a io s I is ecause o is o i ea i y a e i e se o em is ge e a y ea e y o - i ea o imi a io me o s w ic ca ea o so u io s ei g a e i oca mi ima I case o Gaussia se so oise e es es ima o o e eco s uc io qua i y o e sig a is e squa e e o e wee e o ai e a mo e e E/MEG a a
w e e C( q is o e i o uce o egu a i e e so u io i.e., o o ai e esi e ea u es o e es ima e sig a (e.g., smoo ess i ime o i s ace owes e e gy o is e sio a A > 0 is use o a y e a e-o e wee e goo ess o i a e egu a i a io e m
is eas -squa es mo e ca e a ie o e i i i ua ime- oi s ( 1 =
("mo i g i o e" mo e o o a ock ( 1 o a a oi s I e sou ces a e assume
o s ay co s a u i g e ock ( 1 e e so u io wi ime co s a q i( = qi is e a ge O e ea u es e i e om e a a esi es u e E/MEG sig a s as e a gume o (A a ( 1 a e o e use e.g., E /E wa e o ms w ic e ese a e age E/MEG sig a s ac oss mu i e ia s mea ma i e case o s a e
154 155
o e ia / ie o og a y u i g some e io o ime o sig a eque cy com o e s o oca i e e sou ces o e osci a io s o i e es e e i g o e ea me o ( 1 e i e se o em ca e ese e i a cou e o i e e ways e u e- o ce mi imi a io o ( 1 i es ec o o a ame e s a q a e co si e a io o i e e K eu o a sou ces is ge e a y ca e ECD fitting. ecause o o - i ea o imi a io is a oac wo ks o y o cases w e e e e is a e a i e y sma um e o sou ces K, a e e o e e i e se o em o mu a io is o e - e e mi e i.e., (A 1 as o e ac so u io (ε( q I i e ime oca io s o e a ge i o es ca e assume e sea c s ace o o - i ea o imi a io is e uce a e o imi a io ca e s i i o wo s e s (a o - i ea o imi a io o i oca io s o e i o es a e ( a a ysis o e e mi e e s e g o e i o es is assum io co s i u es e so-ca e spatiotemporal ECD model. wo o e amewo ks a e ee sugges e as mea s o a oi i g e i a s associa e wi o - i ea o imi a io is i u e EC ( EC a eam o mi g ese wo a oac es a e ese e i e ai i e e sec io s
B.2 Linear Inverse Methods: Distributed ECD
I case o mu i e simu a eous y ac i e sou ces a a e a i e o so i g e i e se o em y EC i i g is a is i u e sou ce mo e e a e is i u e EC ( EC wi e use u e i e e o e e o is y e o mo e e EC is ase o a s a ia sam i g o e ai o ume a is i u i g e i o es ac oss a ausi e a s a ia y sma a eas w ic cou e a sou ce o eu o a ac i a io I suc cases i e
oca io s ( i a e a ai a e o eac sou ce/ i o e emo i g e ecessi y o o - i ea o imi a io as i e case o e EC i i g e o wa mo e (A 1 ca e ese e o a oise ess case i e ma i o m 15 w e e G M LN lead field ma i is assume o e s a ic i ime e j, i- e y o G esc i es ow muc a se so j is i ue ce y a i o e i w e e j a ies o e a se so s w i e i a ies o e e e y ossi e sou ce o o e mo e s eci ic e e y a is-a ig e com o e o e e y ossi e sou ce co ai sIg,G( Li ī e ec o = i ices o suc o ec io s i.e., I = [i i N, i 2N] w e L = 3 a a = i w e e i o e
as a i e k ow o ie a io Usi g is o a io G ,ī co es o s o e ea ma i o a si g e i o e qi e M T ma i o s e E/MEG a a w i e e LNxT ma i
Q ( o e a Qī = qi ( co es o s o e o ec io s o e EC s mome o o L o ogo a a es
e so u io o ( e ies o i i g a i e se G + o e ma i G o e ess e es ima e Q i e ms o
a wi o uce a i ea ma Q O e a ei g com u a io a y co e ie e e is o muc easo o ake is a oac e ask is o mi imi e e e o u c io ( w ic ca e ge e a i e y e weig i g o e a a o accou o e se so oise a i s co a ia ce s uc u e
w e e W 1 is a weig i g ma i i se so s ace A e o-mea Gaussia sig a ca e c a ac e i e y e si g e co a ia ce ma i
C E I case o a o -si gu a C E e mos sim e weig i g sc eme W C E ca e use o accou o o -u i o m a ossi y co e a e se so oise Suc a u e- o ce a oac so es some o ems o EC mo e i g s eci ica y e equi eme o a o - i ea o imi a io u u o u a e y i i o uces a o e o em e i ea sys em ( is i - ose a u e - e e mi e ecause e um e o sam e ossi e sou ce oca io s is muc ig e a e ime sio a i y o e i u 157 data space (which cannot exceed the number of sensors), i.e., N >> M. Thus, there is an infinite number of solutions for the linear system because any combination of terms from the null space of G will satisfy equation (B.3) and fit the sensor noise perfectly. In other words, many different arrangements of the sources of neural activation within the brain can produce any given MEG or EEG map. To overcome such ambiguity, a regularization term is introduced into the error measure
where A > 0 controls the trade-off between the goodness of fit and the regularization term
C(Q)• The equation (B.5) can have different interpretations depending on the approach used to derive it and the meaning given to the regularization term C(Q). All of the following methods provide the same result under specific conditions [238, 239]: Bayesian methodology to maximize the posterior (Q assuming Gaussian prior on Q
[240], Wiener estimator with proper C, and C s , Tikhonov regularization to trade-off the goodness of fit (B.4) and the regularization term C(Q) = tr(Q WQ-¹Q) which attempts to find the solution with weighted by W -61 minimal 2nd norm, which is usually called Weighted minimum norm (WMN). All the frameworks lead to the solution of the next general form
If and only if WQ and Wx are positive definite [241] (B.6) is equivalent to
In case when viable prior information about the source distribution is available Q , it is easy to account for it by minimizing the deviation of the solution not from 0 (which constitutes the minimal 2nd norm solution G ± ), but from the prior Q , i.e., C(Q) = o e oise ess case wi a weig e - o m egu a i e e Moo e- e ose
seu o-i e se gi es e i e se G + = G y a oi i g e u s ace o ec io s o
G i e so u io us o i i g a u ique so u io wi a mi ima seco o m G = WQG (GWQG -¹
aki g WQ = I W = IM a Q = co s i u es e sim es egu a i e mi imum o m so u io ( ik o o egu a i a io C assica y A is ou usi g c oss- a i a io [ ] o -cu e [ 3] ec iques o eci e ow muc o e oise owe s ou e oug i o e so u io i i s e a [ ] sugges e i e a i e me o eM w e e e co i io a e ec a io o e sou ce is i u io a e egu a i a io a ame e s a e es ima e oi y eM was s ow [ 5] o e o m e e a a egu a WM i e case o i a i oca io io s A i io a co s ai s ca e a e o im ose a a i io a egu a i a io o i s a ce em o a smoo ess [ ] As ese e i ( 7 G+ ca accou o i e e ea u es o e sou ce o
a a s ace y i co o a i g em co es o i g y i o WQ a W e a a- i e ea u es a e commo y use i EMSI
• W C accou s o a y ossi e oise co a ia ce s uc u e o i CE is iago a wi sca e e e o e ms acco i g o e oise e e o eac se so ;
• WQ WC = CS accou s o io k ow e ge o e sou ces co a ia ce s uc u e
WQ ca a so accou o i e e s a ia ea u es
• WQ = W = ( iag (G G - 3 o ma i es e co um s o e ma i G o accou o ee sou ces y e a i i g o e s oo c ose o e se so s [ 7 ] Weig i g eg ee E [0.6 ... ] was s ow o im o e mea oca i a io accu acy [ 9]; 159
• WQ Wgm, w e e e i- iago a e eme i co o a es e g ay ma e co e
i e a ea o e i- i o e [ 5 ] i e e o a i i y o a i g a a ge o u a io o eu o s ca a e o c ea i g e e ec e /MEG sig a ;
• WQ = (Wa Wa - 1 w e e ows o Wa e ese a e agi g coe icie s o eac sou ce [ 51] So a o y geome ica [ 5 ] o io ysica a e agi g ma ices [ 1] we e sugges e ;
• WQ i co o a es e i s -o e s a ia e i a i e o e image [ 53] o a acia o m [ 5 ]
ea u es e i e y e iago a ma ices (e g W a Wgm ca e com i e
oug e sim e ma i o uc A a e a i e a oac is o ese W Q i e ms o a i ea asis se o e i i i ua W Q ac o s i e WQ = A W + A Wg + • • • wi
a e o imi a io o i ia e EM a go i m [ 5 ] o e e co i io e u e - e e mi e i ea i e se o em ( 3 i i s e a [ 5 ] sugges e o e o m e i e se o e a io i e s ace o e a ges eige ec o s o e WQ Suc e ocessi g ca a so e o e i e em o a omai w e a simi a su -s ace se ec io is e o me usi g io em o a co a ia ce ma i us e ec i e y se ec i g e eque cy owe s ec um o e es ima e sou ces Ca e u se ec io o e esc i e ea u es o a a a sou ce s aces e s o im o e e i e i y o e EC so u io e e e ess e i e e am igui y o e i e se so u io ec u es ac ie i g a ig eg ee o oca i a io ecisio I is o is easo a a i io a s a ia i o ma io a ou e sou ce s ace ea i y a ai a e om o e u c io a mo a i ies suc as M I a E ca e o co i io e EC so u io (Sec io 3 160
B.3 Beamforming
eam o mi g (some imes ca e a s a ia i e o a i ua se so is a o e way o so e e i e se o em w ic ac ua y oes o i ec y mi imi e ( 1 A eam o me a em s o i a i ea com i a io o e i u a a qi = i w ic e ese s e
eu o a ac i i y o eac i o e q i i e es ossi e way o e a a gi e ime As i EC me o s e sea c s ace is sam e u i co as o e EC a oac e eam o me oes o y o i a e o se e a a a o ce e i ea y co s ai e mi imum a ia ce ( CM eam o me [ 55] ooks o a s a ia i e e i e as i o si e M L mi imi i g e ou u e e gy i C i u e
e co s ai a o y qi is ac i e a a ime i.e., a e e is o a e ua io o e sig a o i e es k G = δkiI w e e e K o ecke e a Ski = 1 o y i k = i a o e wise ecause e eam o mi g i e i o e i- i o e is e i e i e e e y om e o e ossi e i o es i e i wi e o e om e e i e esu s o e c a i y o ese a io e co s ai e mi imi a io so e usi g ag a ge mu i ie s yie s
is so u io is equi a e o ( w e a ie o a si g e i o e wi e egu a i a io e m omi e Sou ce oca i a io is e o me usi g ( 9 o com u e e a ia ce o e e y i o e q w ic i e case o u co e a e i o e mome s is
e oise-se si i i y o ( 1 ca e e uce y usi g e oise a ia ce o eac i o e
as o ma i i g ac o = ((G ī Cε-¹G ī -¹ is o uces e so-ca e neural activity index 1 1
An alternative beamformer, sy e ic a e u e mag e ome y or SAM [256], is similar to the LCMV if the orientation of the dipole is defined, but it is quite different in the case of a dipole with an arbitrary orientation. A vector of lead coefficients g, (0) is defined
as a function of the dipole orientation. This returns a single vector for the orientation of the i-th dipole, as opposed to the earlier formulation in which the columns of G.,ī played a similar role. With this new formulation, the spatial filter is constructed
which, under standard assumptions, is an optimal linear estimator of the time course of the i-th dipole. The variance of the dipole, accordingly, is also a function of specifically
vg (0) = 1/ (gi(θ C-¹gi(θ o com u e e eu o a ac i i y i e e o igi a SAM
o mu a io uses a s ig y i e e o ma i a io ac o v,(0) = f (0) Cε ( w ic yie s a i e e esu i e oise a ia ce i C is o equa ac oss e se so s
The unknown value of is found via a non-linear optimization of the neuronal activity index for the dipole:
Despite the pitfalls of non-linear optimization, SAM filtering provides a higher SNR to LCMV by bringing less than half of the noise power into the solution. In addition, SAM filtering results in sharper peaks of the distribution of neuronal activity index over the volume [257].
Having computed vg and v, using SAM or LCMV for the two experimental conditions: passive (p) and active (a), it is possible to compute a pseudo-t value t for each location across the two conditions 162
Such an approach provides the possibility of considering experimental design in the analysis of E/MEG localization. Unlike ECD, beamforming does not require prior knowledge of the number of sources, which is a non-trivial problem on its own [258]. Nor beamforming does its search for a solution in an underdetermined linear system as does DECD. For these reasons, beamforming remains the favorite method of many researchers in EMSI and has been suggested for use in the integrative analysis of E/MEG and fMRI which is covered in Section 4.3.5. A E I C
E AI S O E MU IMO A A A A A YSIS
is a e i o i es a i io a i o ma io a ou mu imo a a a ysis o e a a w ic is ese e i Sec io
1 3 1
igu e C 1 responses used for simulation. Blue LFP is the one used for modeling event-related activity, while all the others (including the blue one as well) were arbitrarily assigned to voxels to generate simulated LFP response.
igu e C response used for fMRI data simulation. 1 5
igu e C 3 Gain matrix for EEG signal simulation. 166
Figure C.4 The international 10-20 system seen from (A) left and (B) above the head. A = Ear lobe, C = central, Pg = nasopharyngeal, P = parietal, F = frontal, Fp = frontal polar, 0 = occipital. (C) Location and nomenclature of the intermediate 10% electrodes, as standardized by the American Electroencephalographic Society. (From [2591) 167
igu e C 5 Agg ega e se si i i y o e eque cies i e δ 1-band range of frequencies. 168
Figure C.6 Aggregate sensitivity to the frequencies in the δ - band range of frequencies. 169
igu e C 7 Agg ega e se si i i y o e eque cies i e θ- a a ge o eque cies 17
igu e C Aggregate sensitivity to the signal of CP1 EEG channel (see Figure C.4 for the typical location of the sensor in respect to the brain). 171
Figure C.9 Aggregate sensitivity to the signal of CP2 EEG channel (see Figure C.4 for the typical location of the sensor in respect to the brain). 17
igu e C 1 Aggregate sensitivity to the signal of Pz EEG channel (see Figure C.4 for the typical location of the sensor in respect to the brain). APPENDIX D
FREE OPEN SOURCE SOFTWARE GERMANE TO THE ANALYSIS OF NEUROIMAGING DATA
Table D.1 lists a handful of FOSS projects relevant to the analysis of neuroimaging analysis, whenever Table D.2 lists projects, either written in Python or providing Python bindings, which are germane to acquiring or processing neural information datasets using machine learning (M methods. The last column in Table D.2 indicates whether PyMVPA internally uses a particular project or provides public interfaces to it.
173 bl 1 ee So wa e Ge ma e o e A a ysis o eu oimagi g a a
ai s o m [ ] eu o EM [ 1 ]/ e es io SE [ ]/SCI u [ 3] O e MEEG ai isa/A a omis [ ] eeSu e [ 5] Ca e /Su e i [ ] ai sui e [ 7] EEG/MEG/M I [ ] MEG [ 9] EEG A / M I A [ 7 ] s a s o Input/Output facility for a feature tAn extensive M segmentation bibliography is available online [271] POSIX includes all versions of Unix and GNU/Linux *Matlab Toolbox bl Free and Open-source Projects in Python Related to Neuroimaging Data Acquisition and Processing.
Name Description URL PyMVPA
Mac i e ea i g Elephant Multi-purpose library for ML http://elefant.developer.nicta.com.au Shogun Comprehensive ML toolbox http://www.shogun-toolbox.org/ Orange General-purpose data mining http://www.ailab.si/orange PyML ML in Python http://pyml.sourceforge.net MDP Modular data processing http://mdp-toolkit.sourceforge.net hcluster Agglomerative clustering http://code.google.com/p/scipy-cluster/ Other Python modules http://www.mloss.org/software/language/python eu oscie ce e a e NiPy Neuroimaging data analysis http://neuroimaging.scipy.org PyMGH Access FreeSurfer's mg files http://code.google.com/p/pyfsio PyNIfTI Access NIfTI/Analyze files http://niftilib.sourceforge.net/pynifti OpenMEEG EEG/MEG inverse problems http://www-sop.inria.fr/odyssee/software/OpenMEEG Brian Spiking NN simulation http://brian.di.ens.fr/ PyNN. Specification of NN models http://neuralensemble.org/trac/PyNN S imu i a E e ime esig PyEPL Create complete experiments http://pyepl.sourceforge.net/ VisionEgg Visual Stimuli Generation http://www.visionegg.org PsychoPy Create psychophysical stimuli http://www.psychopy.org/ I Python Imaging Library http://www.pythonware.com/products/pil/ I e aces o O e Com u i g E i o me s RPy Interface to R http://rpy.sourceforge.net/ mlabwrap Interface to Matlab http://mlabwrap.sourceforge.net/ Ge e ic Matplotlib 2D Plotting http://matplotlib.sourceforge.net Mayavi2 Interactive 3D visualization http://code.enthought.com/projects/mayavi PyExcelerator Access MS Excel files http://sourceforge.net/projects/pyexcelerator pywavelets Discrete wavelet transforms http://www.pybytes.com/pywavelets/ E E E CES
[1] C u c a a Se owski The Computational Brain. MI ess Cam i ge MA 199 [ ] Ma k am e ue ai o ec e ISS 1 71- 3 [3] Co e Mag e oe ce a og a y e ec io o e ai s e ec ica ac i i y wi a su e co uc i g mag e ome e Science, 175 - 197 [ ] Y a c e ko S a so a A ea mu e Multimodal Integration: fMRI, MRI, EEG, MEG, c a e ages 3- 5 Sig a ocessi g a Commu ica io s ekke 5 IS 7 5 5 [5] o u ka a A a e i i owa i ec ma i g o eu o a ac i i y M I e ec io o u aweak a sie mag e ic ie c a ges Magn Reson Med, 7( 1 5 -1 5 [ ] io g o a Gao i ec y ma i g mag e ic ie e ec s o eu o a ac i i y y mag e ic eso a ce imagi g Hum Brain Mapp, (1 1- 9 3 [7] C S oy a C S S e i g o O e egu a io o e oo -su y o e ai J Physiol (London), 11 5-1 1 9 [ ] S Ogawa ee A ayak a G y O yge a io -se si i e co as i mag e ic eso a ce image o o e ai a ig mag e ic ie s Magn Reson Med, 1 (1 -7 199 [9] S Ogawa M ee A Kay a W a k ai mag e ic eso a ce imagi g wi co as e e e o oo o yge a io Proc Natl Acad Sci U S A 7 ( 9 -9 7 199 [1 ] e i eau Ke e y McKi s y uc i e Weissko M Co e e ea a y a ose u c io a ma i g o e uma isua co e y mag e ic eso a ce imagi g Science, 5 (5 3 71 -719 1991 [11] ose e i eau A o e Ke e y uc i e A isc ma M G u e G as Weissko M Co e a e a Susce i i i y co as imagi g o ce e a oo o ume uma e e ie ce Magn Reson Med, ( 93-9; iscussio 3 -3 1991 [1 ] S ackowiak K is o o a a C ice Human Brain Function. Aca emic ess seco e i io o em e 3 IS 1 1
17 177
[13] A O oo e ia g A i e a u o a M A a e eo e ica s a is ica a ac ica e s ec i es o a e - ase c assi ica io a oac es o e a a ysis o u c io a eu oimagi g a a Journal of Cognitive Neuroscience, 19 1735-175 7 [1 ] K iegesko e a a e i i A a y i g o i o ma io o ac i a io o e oi ig - eso u io M I Neurolmage, 3 ( 9- ec 7 ISS 1 53- 119 [15] e ei a Mi c e a M o i ick Mac i e ea i g c assi ie s a m i A u o ia o e iew Neurolmage, i ess [1 ] S So e u g M au C S O g S e gio o ou G o mes Y eCu K - Wi e e ei a C E asmusse G ä sc Sc ö ko A Smo a i ce Wes o a Wi iamso e ee o o e sou ce so wa e i mac i e ea i g Journal of Machine Learning Research, 3- 7 [17] a ik The Nature of Statistical Learning Theory. S i ge ew Yo k 1995 IS -3 7-9 559- [1 ] Y Kami a i a o g eco i g e isua a su ec i e co e s o e uma ai Nature Neuroscience, 79- 5 5 ISS 1 97- 5 [19] - ay es K Sakai G ees S Gi e C i a E assi g am ea i g i e i e io s i e uma ai Current Biology, 17 3 3-3 7 ISS 9 -9 [ ] K iegesko e Goe e a a e i i I o ma io - ase u c io a ai ma i g Proceedings of the National Academy of Sciences of the USA, 1 3 3 3-3 ISS 7- [ 1] K A o ma S M o y G e e a a y eyo mi - ea i g mu i- o e a e a a ysis o M I a a Trends in Cognitive Science, 1 - 3 ISS 13 - 13 [ ] M a ke Y a c e ko Se e e g S a so a y a S o ma yM A A y o oo o o mu i a ia e a e a a ysis o M I a a Neuroinformatics, 7 (1): 37-53 Ma 9 [ 3] M a ke Y a c e ko Se e e g E O i e i I ü W iege C S e ma a y S a so a S o ma yM A A u i yi g a oac o e a a ysis o eu oscie i ic a a Frontiers in Neuroinformatics, 3(3 9 [ ] - ay es a G ees e ic i g e o ie a io o i isi e s imu i om ac i i y i uma ima y co e Nature Neuroscience, - 91 5 17
[ 5] a y M Go i i M u ey A Is ai Sc ou e a ie i i is i u e a o e a i g e ese a io s o aces a o ec s i e a em o a co e Science, 93 5- 3 1
[ ] S a so Ma suka a a y Com i a o ia co es i e a em o a o e o o ec ecog i io a y ( 1 e isi e is e e a " ace" a ea? Neurolmage, 3 15 -1 [ 7] A O oo e ia g A i a a y a ia y is i u e e ese a io s o o ec s a aces i e a em o a co e Journal of Cognitive Neuroscience, 17 5 -59 5
[ ] S a so a Y a c e ko ai ea i g usi g u ai su o ec o mac i es o o ec ecog i io e e is o " ace" i e i ica io a ea Neural Computation, -5 3 ISS 99-7 7 [ 9] M a ke Y a c e ko Se e e g a M ug es The PyMVPA Manual. A ai a e o i e a //www ym a o g/ yM A-Ma ua
[3 ] G e e S M o y C Moo e a u Si ge Co e a y a K A o ma e mu i- o e a e a a ysis (M A oo o os e ese e a e A ua Mee i g o e O ga i a io o uma ai Ma i g ( o e ce I a y [31] S aCo e S S o e C e kassky A e so a u Su o ec o mac i es o em o a c assi ica io o ock esig M I a a Neurolmage, 317-3 9 5 ISS 1 53- 119 [3 ] C eckma a S M Smi e so ia e e sio s o i e e e com o e a a ysis o mu isu ec M I a a ysis NeuroImage, 5(1 9 -311 Ma 5 ISS 1 53- 119 [33] S Makeig S e e e O o a A e o me Mi i g e e - e a e ai y amics Trends Cogn Sci, (5 -1 May ISS 13 - 13 [3 ] Y Su I e a i e E IE o ea u e weig i g A go i ms eo ies a a ica io s IEEE Transactions on Pattern Analysis and Machine Intelligence, 9( 1 35-1 51 u e 7
[35] I Guyo Wes o S a i a a ik Ge e se ec io o ca ce c assi ica io usi g su o ec o mac i es Machine Learning, 3 9- ISS 5- 1 5 [3 ] I Guyo a A E issee A i o uc io o a ia e a ea u e se ec io Journal of Machine Learning, 3 1157-11 3 [37] K is a u am Ca i M A iguei e o a A a emi k S a se mu i omia ogis ic eg essio as a go i ms a ge e a i a io ou s 179
IEEE Transactions on Pattern Analysis and Machine Intelligence, 7 957- 9 5 [3 ] E o a i s i a i An Introduction to the Bootstrap. C a ma a /C C 1993 [39] E ic o s a A o mes o a ame ic e mu a io es s o u c io a eu oimagi g A ime wi e am es Human Brain Mapping, 15 1- 5 1 [ ] C e e ei a W ee S S o e a Mi c e E o i g e ic i e a e o uci e mo e i g wi e si g e-su ec IAC a ase Human Brain Mapping, 7 5 - 1 [ 1] A ako omamo y a ia e se ec io usi g S M- ase c i e ia Journal of Machine Learning Research, 3 1357-137 3 [ ] S Sa y a Ke y O ga i a io o au i o y co e i e a i o a sou eque cy J Neurophysiol, 59(5 1 7-3 May 19 ISS -3 77 [ 3] C a y e i o MI ess Cam i ge MA Oc IS 3337 [ ] - u g S Makeig M Wes e ie ow se E Cou c es e a Se owski A a y i g a isua i i g si g e- ia e e - e a e o e ia s I Adv in Neu Info Proc Sys 11, ages 11 - MI ess 1999 [ 5] A C a g A ea mu e u g a A Ma as e ko I e e e com o e s o mag e oe ce a og a y Si g e- ia es o se o se imes Neurolmage, [ ] A ea mu e A C a g M i u e sky A e y a M Weise A MEG s u y o es o se a e cy a a ia i i y i e uma isua sys em u i g a sim e isua -mo o i eg a io ask Soc for Neurosci Abstracts, 9 (7 1999 [ 7] A ea mu e S a ami o A C a g a G o e e a io s i e wee e ase o a a- a osci a io a si g e- ia isua y e oke es o ses Soc for Neurosci Abstracts, 31( 1 1 1 [ ] a a C A i o A a g ea mu e Yeu g A Osma a Sa a i ea s a ia i eg a io o si g e- ia e ec io i e ce a og a y Neurolmage, 17(1 3-3 Se [ 9] I ü A usc Sc a ow G u e U Kö e a C S e ma ime essu e mo u a es e ec o ysio ogica co e a es o ea y isua ocessi g PLoS ONE, 3( e1 75 ISS 193 - 3 1
[5 ] A usc C S e ma M M Mü e e a G u e A c oss- a o a o y s u y o e e - e a e gamma ac i i y i a s a a o ec ecog i io a a igm Neurolmage, 33 11 9-1177 [51] C S e ma e S u ge A usc a Maess Memo y-ma c es e oke uma gamma- es o ses BMC Neurosci, 5(13 [5 ] C E asmusse a C K Wi iams Gaussian Processes for Machine Learning. MI ess Cam i ge MA [53] - ac au Geo ge C a o - au y Ma i e ie ugue i e Mi o i Ka a e a e au e ma y aces o e gamma a es o se o com e isua s imu i Neurolmage, 5 91-5 1 5 [5 ] W iege C eic e K Gege u e oesse C au - ei e K use a i ic s e ic i g e ecog i io o a u a sce es om si g e ia MEG eco i gs o ai ac i i y Neurolmage, (3 1 5 - Se ISS 1 95-957 [55] W iege C au ü o a K Gege u e e y amics o isua a e maski g i a u a sce e ocessi g A mag e oe ce a og a y s u y J. Ws., 5(3 75- 3 5 ISS 153 -73 [5 ] K e o ou os C Cam e a C is ia i i Co o i g e se si i i y o su o ec o mac i es I Proceedings of the International Joint Conference on AI, ages 55- 1999 [57] Ka wis e Mc e mo a M M C u e usi o m ace a ea a mo u e i uma e as ia e co e s ecia i e o ace e ce io J Neurosci, 17 (11 3 -11 1997 [5 ] Ka wis e S a ey a A a is e usi o m ace a ea is se ec i e o aces o a ima s Neuroreport, 1 (1 1 3-7 1999 [59] M S i i o a Ka wis e ow is i u e is isua ca ego y i o ma io i uma occi i o- em o a co e ? A M I s u y Neuron, 35( 1157- 5 [ ] Ka wis e eu oscie ce W a s i a ace? Science, 311(57 1 17- [ 1] I Gau ie Sku a ski C Go e a A W A e so E e ise o ca s a i s ec ui s ai a eas i o e i ace ecog i io Nat Neurosci, 3( 191-7 [ ] K a uce Su es a A e sky Foundations of Measurement, o ume 1 Aca emic ess Sa iego 1971 [ 3] uce Individual choice behavior: a theoretical analysis. Wi ey Y 1959 181
[ ] K G i -S ec o a Ka wis e isua ecog i io as soo as you k ow i is e e you k ow w a i is Psychol Sci, 1 ( 15 - 5 [ 5] I Guyo Wes o S a i a a ik Ge e se ec io o ca ce c assi ica io usi g su o ec o mac i es Mach Learn, (1-3 3 9- ISS 5- 1 5 [ ] S a so a M y e is i u io o O susce i i i y e ec s i e ai is o -Gaussia Neuroreport, 1 (9 1971-7 1 [ 7] C C C e C W y e a A ase e S a is ica o e ies o O mag e ic eso a ce ac i i y i e uma ai NeuroImage, ( 1 9 -1 9 3 [ ] A M Wi k a M oe i k O oise assum io s i M I Mt. J. Biomed. Imaging, [ 9] S a so a u W a co ec io is mo e s ea owa a eo y o e ese a io i co ec io is e wo ks Behavioral and Brain Sciences, 13 71-51 199 [7 ] Mi c e u c i so S icu escu e ei a Wa g M us a S ewma ea i g o eco e cog i i e s a es om ai images Machine Learning, 57 1 5-175 [71] C ao A Ma i a a y A e ace- es o si e egio s se ec i e o y o aces? Neuroreport, 1 (1 9 5-5 1999 [7 ] I Gau ie M a Moy a Sku a ski C Go e a A W A e so e usi o m " ace a ea" is a o a e wo k a ocesses aces a e i i i ua e e J Cogn Neurosci, 1 (3 95-5 [73] Ma ac I e y a U asso e o og a y o ig -o e uma o ec a eas Trends Cogn Sci, ( 17 -1 [7 ] M e ki so a is e a y a S Smi Im o e o imisa io o e o us a accu a e i ea egis a io a mo io co ec io o ai images Neurolmage, 17:825-841, [75] i ey M We s e a e au e Sei ma Go s ei o esi as Gu ie e S Eick o K Amu s K i es a cas e C ase g o e Ke e y M e ki so a S Smi A a omica ai a ases a ei a ica io i e S iew isua isa io oo I Thirteenth Annual Meeting of the Organization for Human Brain Mapping, 7 [7 ] S M Smi M e ki so M W Woo ic C eckma E e e s o a se - e g a is e M e uca I o ak E i ey K ia y Sau e s icke s Y a g e S e a o M a y a M Ma ews A a ces i u c io a a s uc u a M image a a ysis 1
a im eme a io as S Neurolmage, 3 Su 1 S -19 ISS 1 53- 119 [77] I Guyo ose a a ik Au oma ic ca aci y u i g o e y a ge C- ime sio c assi ie s I S a so Cowa a C Gi es e i o s Advances in Neural Information Processing Systems, o ume 5 ages 1 7- 155 Mo ga Kau ma Sa Ma eo CA 1993 [7 ] K is o o s ei Ge g S e e a e so A c i ique o u c io a oca ise s Neurolmage, 3 ( 1 77- 7 [79] Sa e M e a Ka wis e i i e a co que a e e se o u c io a oca i e s Neurolmage, 3 ( 1 -9 ; iscussio 1 97-9 [ ] Is ak a G a as A e icie me o o a ia e se ec io usi g S M ase c i e ia Journal of Machine Learning Research, 5 [ 1] y ma a Y a Sam e qua i es i s a is ica ackages The American Statistician, 5 ( 3 1-3 199 [ ] a y E A o ma a M I Go i i e is i u e uma eu a sys em o ace e ce io Trends Cogn Sci, ( 3- 33 [ 3] a ey as I imi e a ke S e ig a M Seeck u c io a M I wi simu a eous EEG eco i g easi i i y a a ica io o mo o a isua ac i a io J Magn Reson Imaging, 13( 9 3-9 1 [ ] K Si g G a es A i e a E M o e a A Wi iams ask- e a e c a ges i co ica sy c o i a io a e s a ia y coi ci e wi e emo y amic es o se Neurolmage, 1 (1 1 3-11 [ 5] M Somme Mei a a o Com i e measu eme o e e - e a e o e ia s (E s a M I Acta Neurobiol Exp (Wars), 3(1 9-53 3 [ ] S a i Wa ki g M o a C e o -Ma i u ie a C Sege a Seque ce o a e o se es o ses i e uma isua a eas a M I co s ai e E sou ce a a ysis Neurolmage, 1(3 1- 17 [ 7] K W i i gs a G S oi k a M Sc mi E a ua i g e s a ia e a io s i o e e - e a e o e ia a u c io a M I sou ces i e ima y isua co e Hum Brain Mapp, [ ] Y agai C i c ey E ea e s o e e wick M im e a o a ai ac i i y e a i g o e co i ge ega i e a ia io a M I i es iga io Neurolmage, 1( 1 3 - 1 [ 9] A Ko e o a u u e E Sa i o o e S Ma i kau i M a a au o e i a e I mo iemi a A o e Ac i a io o mu i e co ica a eas i es o se o soma ose so y s imu a io com i e 183
magnetoencephalographic and functional magnetic resonance imaging. Hum Brain Mapp, 8(1):13-27, 1999. [90] M. Schulz, W. Chau, S. J. Graham, A. R. McIntosh, B. Ross, R. Ishii, and C. Pantev. An integrative MEG-fMRI study of the primary somatosensory cortex using cross-modal correspondence analysis. Neurolmage, 22(1):120-133, 2004. [91] M. S. Cohen, R. I. Goldman, J. Stern, and J. Engel, Jr. Simultaneous EEG and fMRI made easy. Neurolmage, 13(6 Supp.1):S6, Jan. 2001. [92] R. I. Goldman, J. M. Stern, J. Engel, Jr, and M. S. Cohen. Simultaneous EEG and fMRI of the alpha rhythm. Neuroreport, 13(18):2487-2492, 2002. [93] M. Moosmann, P. Ritter, I. Krastel, A. Brink, S. Thees, F. Blankenburg, B. Taskin, H. Obrig, and A. Villringer. Correlates of alpha rhythm in functional magnetic resonance imaging and near infrared spectroscopy. Neurolmage, 20(1):145- 158, 2003. [94] H. Laufs, A. Kleinschmidt, A. Beyerle, E. Eger, A. Salek-Haddadi, C. Preibisch, and K. Krakow. EEG-correlated fMRI of human alpha activity. Neurolmage, 19(4):1463-1476, 2003. [95] M. J. Makiranta, J. Ruohonen, K. Suominen, E. Sonkajarvi, T. Salomaki, V. Kiviniemi, T. Seppanen, S. Alahuhta, V. Jantti, and 0. Tervonen. BOLD- contrast functional MRI signal changes related to intermittent rhythmic delta activity in EEG during voluntary hyperventilation-simultaneous EEG and fMRI study. Neurolmage, 22(1):222-231, 2004. [96] J. Foucher, H. Otzenberger, and D. Gounot. Where arousal meets attention: a simultaneous fMRI and EEG recording study. Neurolmage, 22(2):688-697, 2004. [97] S. G. Horovitz, P. Skudlarski, and J. C. Gore. Correlations and dissociations between BOLD signal and P300 amplitude in an auditory oddball task: a parametric approach to combining fMRI and ERP. Magn Reson Imaging, 20 (4):319-325, 2002. [98] M. Brazdil, M. Dobsik, M. Mikl, P. Hlustik, P. Daniel, M. Pazourkova, P. Krupa, and I. Rektor. Combined event-related fMRI and intracerebral ERP study of an auditory oddball task. Neurolmage, 26(1):285-93, 2005. [99] T. Eichele, K. Specht, M. Moosmann, M. L. Jongsma, R. Q. Quiroga, H. Nordby, and K. Hugdahl. Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Nati Acad Sci U S A, 102(49):17798-803, 2005. [100] E. Liebenthal, M. L. Ellingson, M. V. Spanaki, T. E. Prieto, K. M. Ropella, and J. R. Binder. Simultaneous ERP and fMRI of the auditory cortex in a passive oddball paradigm. Neurolmage, 19(4):1395-1404, 2003. 1
[1 1] K ugge C S e ma C Wiggi s a Y o C amo emo y amic a e ec oe ce a og a ic es o ses o i uso y igu es eco i g o e e oke o e ia s u i g u c io a M I Neurolmage, 1 ( 13 7-133 1 [1 ] Se a ou S Mo o m a i a o e S a io em o a y amics o uma o ec ecog i io ocessi g A i eg a e ig - e si y e ec ica ma i g a u c io a imagi g s u y o "c osu e" ocesses Neurolmage, 5 [1 3] Me o M o K im G G o e a A e e aum Com i e e e - e a e M I a EEG e i e ce o em o a - a ie a co e ac i a io u i g a ge e ec io Neuroreport, (1 3 9-3 37 1997 [1 ] C Mu e age Sc mi uss e oga e Mo e G ucke a U ege I eg a io o M I a simu a eous EEG owa s a com e e si e u e s a i g o oca i a io a ime-cou se o ai ac i i y i a ge e ec io Neurolmage, (1 3-9 [1 5] S G o o i ossio Sku a ski a C Go e a ame ic esig a co e a io a a a yses e i eg a i g M I a e ec o ysio ogica a a u i g ace ocessi g Neurolmage, ( 15 7-1595 [1 ] ua g- e i ge C ei e G McCo mack M S Co e K K Kwo g Su o Sa oy M Weissko a is ake W e i eau a ose Simu a eous u c io a mag e ic eso a ce imagi g a e ec o ysio ogica eco i g Hum Brain Mapp, 3 13- 5 1995 [1 7] i acco a eis ascua -Ma qui a E Ma i Co es o e ce o e e - e a e o e ia omog a y a u c io a mag e ic eso a ce imagi g u i g a guage ocessi g Hum Brain Mapp, 17(1 -1 [1 ] G ummic C imsky E au i M uc e e a Ga s a Com i i g M I a MEG i c eases e e ia i i y o esu gica a guage oca i a io A c i ica s u y o e i e e ce e wee a co g ue ce o o mo a i ies Neurolmage, [1 9] i e e ge Weisko ei Wi e m E e a a i aume A EEG- i e ai -com u e i e ace com i e wi u c io a mag e ic eso a ce imagi g ( M I IEEE Trans Biomed Eng, 51( 971- [11 ] S Wa ac I es G Sc aug M a e G a y a ga a E e ma a Sc ome EEG- igge e ec o- a a u c io a M I i e i e sy Neurology, 7(1 9-93 199 1 5
[111] M Seeck a ey as C M Mic e a ke C A Ge icke I es e a e e Go ay C A ae gge i e i o e a a is o - i asi e e i e ic ocus oca i a io usi g EEG- igge e u c io a M I a e ec omag e ic omog a y Electroencephalogr Clin Neurophysiol, 1 ( 5 -51 199 [11 ] K K akow U C Wies ma G Woe ma M Symms M A Mc ea emieu A e G a ke is a S u ca Mu imo a M imagi g u c io a i usio e so a c emica s i imagi g i a a ie wi oca i a io - e a e e i e sy Epilepsia, (1 1 59-1 1999 [113] K K akow G Woe ma M Symms A e emieu G a ke S u ca a is EEG- igge e u c io a M I o i e ic a e i e i o m ac i i y i a ie s wi a ia sei u es Brain, 1 (9 1 79- 1 1999 [11 ] K K akow emieu Messi a C A Sco M Symms S u ca a is S a io- em o a imagi g o oca i e ic a e i e i o m ac i i y usi g EEG- igge e u c io a M I Epileptic Disord, 3( 7-7 1 [115] G a S i e i G Me e e M Seeck a C M Mic e oca i a io o is i u e sou ces a com a iso wi u c io a M I Epileptic Disord, S ecia Issue 5-5 1 [11 ] emieu K K akow a is Com a iso o s ike- igge e u c io a M I O ac i a io a EEG i o e mo e oca i a io Neurolmage, 1 (5 1 97-11 1 [117] A Wai es M S aw ie ma A a a e A o a G ackso ow e ia e a e M I-EEG s u ies o e i e sy? A o a ame ic a oac o a a ysis a i a io a o imi a io Neurolmage, (1 19 -199 5 [11 ] A is o C e Mu ck K ama i au s Osse ok S u ca a emieu A a ysis o EEG- M I a a i oca e i e sy ase o au oma e s ike c assi ica io a Sig a S ace o ec io Neurolmage, [119] Y u C G o a E Ko ayas i u eau a Go ma Usi g o e -s eci ic emo y amic es o se u c io i EEG- M I a a a a ysis A es ima io a e ec io mo e Neurolmage, 3 (1 195- 3 7 [1 ] S M Mi sa a i Wa g I es i a i S eu g a a a S Me o i ea as ec s o a s o ma io om i e ic a e i e ic isc a ges o O M I sig a s i a a ima mo e o occi i a e i e sy Neurolmage, 3 ( 1133-11 [1 1] i e a A i i ge Simu a eous EEG- M I Neurosci Biobehav Rev, 3 ( 3-3 1
[1 ] G Samme C ecke Ge a Ki sc S a k a ai Acquisi io o y ica EEG wa e o ms u i g M I SS E a o a e a Neurolmage, ( 1 1 -1 5 [1 3] K a se A ie se S C S o e a a ge Co se sus i e e ce i eu oimagi g Neurolmage, 13( 1 1 - 1 [1 ] C Im iu a g W C e a e u c io a co ica sou ce imagi g om simu a eous y eco e E a M I J Neurosci Methods, [1 5] au s au i eau W Ca mic ae a A K ei sc mi ece a a ces i eco i g e ec o ysio ogica a a simu a eous y wi mag e ic eso a ce imagi g Neurolmage, 7 [1 ] I es S Wa ac Sc mi E e ma a Sc ome Mo i o i g e a ie s EEG u i g ec o a a M I Electroencephalogr Clin Neurophysiol, 7( 17- 1993 [1 7] M C Sc mi A Oe e ma C uc em K ogo e is a S M Smi akis Simu a eous EEG a M I i e macaque mo key a 7 es a Magn Reson Imaging, ( 335- [1 ] A e ose s a u e A me o o emo i g imagi g a i ac om co i uous EEG eco e u i g u c io a M I Neurolmage, 1 ( 3 - 39 [1 9] A Sa ek- a a i M Me sc emke emieu a is Simu a eous EEG-Co e a e Ic a M I Neurolmage, (1 3 - [13 ] M S Co e Me o a a a a us o e uci g co ami a io o a e ec ica sig a U i e S a es a e a ica io 97 [131] K A ami Mo i a aka Y Kawagoe Okamo o M Ya i a O is i M Yumo o Ma su a a Sai o S e i g s o e sam i g o e ie i g a i ac - ee e ec oe ce a og am u i g u c io a mag e ic eso a ce imagi g Neurolmage, 19( 1 1- 95 3 [13 ] Ma e kow a e oesige a a eis Sy c o i a io aci i a es emo a o M I a e ac s om co cu e EEG eco i gs a i c eases usa e a wi Neurolmage, 3 (3 11 - [133] S I Go ca es ouwe s Kui e M ee aa a C e Mu ck A i ac emo a i co- egis e e EEG/ M I y se ec i e a e age su ac io Clin Neurophysiol, 7 [13 ] I Go ma M S e E ge a M S Co e Acqui i g simu a eous EEG a u c io a M I Clin Neurophysiol, 111(11 197 -19 1 7
[135] M egis i M A i gaa i o a o Co s a e emo a o ime- a yi g g a ie a i ac s om EEG a a acqui e u i g co i uous M I Clin Neurophysiol, 115(9 1 1- 19 [13 ] K ia y C eckma G Ia e i M a y a S M Smi emo a o M I e i o me a i ac s om EEG a a usi g o ima asis se s Neurolmage, (3 7 -37 5 [137] O e e ge Gou o a ouc e O imisa io o a os - ocessi g me o o emo e e u se a i ac om EEG a a eco e u i g M I a a ica io o 3 eco i gs u i g e- M I Neurosci Res, 57( 3 -9 7 [13 ] Si e s I Mic ie s M e oye a Au eke ke A a e i e a a yck es o a io o M -i uce a i ac s i simu a eous y eco e M /EEG a a Magn Reson Imaging, 17(9 13 3-1391 1999 [139] G o massa u o I aaske ai e K C ia a So o E ow a W e i eau Mo io a a is oca iog am a i ac emo a o i e ea e eco i g o EEG a E s u i g M I Neurolmage, 1 ( 11 7-11 1 [1 ] G Ga e a M Ca i G Gua ie a G icci o ao e Ca i Mo asso a a o C Co o ese oma a Ma a ig ia ea - ime M a i ac s i e i g u i g co i uous EEG/ M I acquisi io Magn Reson Imaging, 1(1 1175-11 9 3 [1 1] G S i as a a S C o a - e e e K au G G o e a Me o ICA- ase oce u es o emo i g a is oca iog am a i ac s om EEG a a acqui e i e M I sca e Neurolmage, (1 5 - 5 [1 ] K Kim W Yoo a W a k Im o e a is oca iac a i ac emo a om e e ec oe ce a og am eco e i M I J Neurosci Methods, 135 (1- 193- 3 [1 3] A i K C ia a ua g- e i ge a G e ki s EEG u i g M imagi g i e e ia io o mo eme a i ac om a o ysma co ica ac i i y Neurology, 5 (1 19 -19 3 1995 [1 ] K ugge C Wiggi s C S e ma a Y o C amo eco i g o e e e - e a e o e ia s u i g u c io a M I a 3 es a ie s e g Magn Reson Med, ( 77- [1 5] S e e e A S o e So ge e e s C K a c ioc A K E ge a Goe e Im o e qua i y o au i o y e e - e a e o e ia s eco e simu a eous y wi 3- M I emo a o e a is oca iog am a e ac Neurolmage, 3 ( 5 7-97 7 1
[1 ] M Si ia c ki Moe e aco s U S e a i oo S Wo a se Sie e a M Sc e g S a ia i e s a au oma e s ike e ec io ase o ai o og a ies im o e se si i i y o EEG- M I s u ies i oca e i e sy Neurolmage, 37(3 3 - 3 7 [1 7] emieu A e a co i M Symms a is eco i g o EEG u i g M I e e ime s a ie sa e y Magn Reson Med, 3 ( 9 3-95 1997 [1 ] K K akow A e M Symms emieu ose s a is EEG eco i g u i g M I e e ime s image qua i y Hum Brain Mapp, 1 (1 1 -15 [1 9] M A ge o e A o as Sego e S Iwaki W e i eau a G o massa Me a ic e ec o es a ea s i simu a eous EEG-M I s eci ic a so io a e (SA simu a io s u ies Bioelectromagnetics, 5 ( 5- 95 [15 ] G o massa Sc wa A K iu K K Kwo g A M a e a W e i eau S a io em o a ai imagi g o isua -e oke ac i i y usi g i e ea e EEG a M I eco i gs Neurolmage, 13( 1 1 35-1 3 1 [151] a ey as O a ke S e ig I imi e Go ay e a e e C M Mic e e i o e G i emu e a M Seeck EEG- igge e u c io a M I i a ie s wi a maco esis a e i e sy J Magn Reson Imaging, 1 (1 177-1 5 [15 ] Sc ome G o massa a ey as M Seeck A um K A ami Sc wa W e i eau a I es EEG- i ke u c io a mag e ic eso a ce imagi g i e i e sy a cog i i e eu o ysio ogy J Clin Neurophysiol, 17(1 3-5 [153] S Ogawa S Me o W a k S G Kim Me k e M E e ma a K Ugu i u c io a ai ma i g y oo o yge a io e e - e e e co as mag e ic eso a ce imagi g a com a iso o sig a c a ac e is ics wi a io ysica mo e Biophysical Journal, 1993 [15 ] u o a a k A mo e o e cou i g e wee ce e a oo ow a o yge me a o ism u i g eu a s imu a io J Cereb Blood Flow Metab, 17( -7 1997 [155] O a a iu K Mi e W u E Wo g a k a u o isc e a cies e wee O a ow y amics i ima y a su eme a y mo o a eas a ica io o e a oo mo e o e i e e a io o O a sie s Neurolmage, 1(1 1 -53 189
[15 ] K E S e a Weisko M ys a e A o i so a K is o Com a i g emo y amic mo e s wi CM Neurolmage, 7 [157] A Seiyama Seki C a a e I Sase A aka suki S Miyauc i E a S ayas i Ima uoka Iwaku a a Ya agi a Ci cu a o y asis o M I sig a s e a io s i e wee c a ges i e emo y amic a ame e s a O sig a i e si y NeuroImage, 1( 1 -1 1 [15 ] e eu a auge as Usi g o i ea mo e s i M I a a a a ysis Mo e se ec io a ac i a io e ec io Neurolmage, 3 ( 1 9- 9 [159] Ki e Ma ou e so a K is o emo y amic co e a es o EEG A eu is ic Neurolmage, (1 - 5 [1 ] K is o e a a u e A a ysis o u c io a M I ime-se ies Hum Brain Mapp, 1 153-171 199 [1 1] G M oy o S A E ge G G o e a eege i ea sys ems a a ysis o u c io a mag e ic eso a ce imagi g i uma 1 J Neurosci, 1 (13 7- 1 199 [1 ] M S Co e a ame ic a a ysis o M I a a usi g i ea sys ems me o s Neurolmage, ( 93-1 3 1997 [1 3] a ge a S ege o - i ea ou ie ime se ies a a ysis o uma ai ma i g y u c io a mag e ic eso a ce imagi g Appl Stat, (1 1- 9 1997 [1 ] G G o e eco o u io o im u se es o se i e e - e a e O M I Neurolmage, 9( 1 - 9 1999 [1 5] C a a akse K ugge M Maisog a Y o C amo Mo e i g emo y amic es o se o a a ysis o u c io a M I ime-se ies Hum Brain Mapp, ( 3-3 199 [1 ] Ciuciu o i e G Ma e ec I ie C a ie a e a i U su e ise o us o a ame ic es ima io o e emo y amic es o se u c io o a y M I e e ime IEEE Trans Med Imaging, (1 1 35-1 51 3 [1 7] Gi e ma W e y As u e a K is o Mo e i g egio a a syc o ysio ogic i e ac io s i M I e im o a ce o emo y amic eco o u io Neurolmage, 19(1 - 7 3 [1 ] G Ma e ec e a i Ciuciu M e eg i i-Issac a o i e o us ayesia es ima io o e emo y amic es o se u c io i e e - e a e O M I usi g asic ysio ogica i o ma io Hum Brain Mapp, 19(1 1-17 3 19
[1 9] Y u A ags aw C G o a E Ko ayas i u eau a Go ma Usi g o e -s eci ic emo y amic es o se u c io i EEG- M I a a a a ysis Neurolmage, 3 (1 3 - 7 [17 ] A So ysik K K eck K W i e C osso a W iggs Com a iso o emo y amic es o se o i ea i y ac oss ima y co ica a eas Neurolmage, (3 1117-11 7 [171] A we a C Ia eco a e eu a asis o u c io a ai imagi g sig a s Trends Neurosci, 5(1 1- 5 [17 ] u e a Si e s ei O e e a io s i o sy a ic ac i i y o mac osco ic measu eme s oes co- egis a io o EEG wi M I make se se? Brain Topogr, 13( 79-9 [173] K ogo e is a A Wa e I e e i g e O sig a Annu Rev Physiol, 735-7 9 [17 ] J. A u s a S o i ace ow we o we u e s a e eu a o igi s o e M I O sig a ? Trends Neurosci, 5(1 7-31 [175] Wa ie a K Iwa a M aka as i Waka ayas i a Kawas ima e eu a asis o e emo y amic es o se o i ea i y i uma ima y isua co e Im ica io s o eu o ascu a cou i g mec a ism NeuroImage, [17 ] K is o ayesia es ima io o y amica sys ems a a ica io o M I Neurolmage, 1 ( 513-3 [177] K ogo e is au s M Auga i a a A Oe e a eu o ysio ogica i es iga io o e asis o e M I sig a Nature, 1 ( 3 15 -157 1 [17 ] S Kim I o e C O ma S G Kim K Ugu i a o S a ia e a io s i e wee eu o a ac i i y a O u c io a M I Neurolmage, 1(3 7 - 5 [179] u e oge S G iamo M A a cesc i i a A oas A em o a com a iso o O AS a I S emo y amic es o ses o mo o s imu i i a u uma s Neurolmage, 9( 3 - [1 ] A e o A K u M A e ma I U e A oas a A M a e Cou i g o o a emog o i co ce a io o yge a io a eu a ac i i y i a soma ose so y co e Neuron, 39( 353-359 3 [1 1] S au esu S ackowiak a G o i i Ma s o soma ose so y sys ems I S J. ackowiak e i o Human brain function, age 5 Aca emic ess Sa iego CA 1997 191
[1 ] K C oi Y I C e E ame a G e ki s ai emo y amic c a ges me ia e y o ami e ece o s o e o e ce e a mic o ascu a u e i o ami e-me ia e eu o ascu a cou i g Neurolmage, [1 3] S e a o ic M Wa ki g a G ike emo y amic a me a o ic es o ses o eu o a i i i io Neurolmage, ( 771-77 [1 ] ai A ou ei g O Brain Res Brain Res Rev, 5 [1 5] ie A Sa ek- a a i is a emieu Ma i g o s ikes s ow wa es a mo o asks i a a ie wi ma o ma io o co ica e e o me usi g simu a eous EEG a M I Magn Reson Imaging, 1(1 11 7-73 3
[1 ] E ie e meye A a y ms as ysio ogica a a o ma e ome a Int J Psychophysiol, (1-3 31- 9 1997 [1 7] S Go ca es e Mu ck ouwe s Sc oo o e Kui e Mau i s oog ui E a Some e ee aa a o es a Si a Co e a i g e a a y m o O usi g simu a eous EEG/ M I I e -su ec a ia i i y Neurolmage, 5 [1 ] au s K K akow S e e E Ege A eye e A Sa ek- a a i a A K ei sc mi E ec oe ce a og a ic sig a u es o a e io a a cog i i e e au mo es i s o a eous ai ac i i y uc ua io s a es Proc Natl Acad Sci US A 1 (19 11 53-11 5 3 [1 9] C e Mu ck S I Go ca es ui oom Kui e ouwe s M ee aa a o es a Si a e emo y amic es o se o e a a y m a EEG/ M I s u y Neurolmage, 35(3 11 -51 7 [19 ] E Ma i e -Mo es A a es-Sosa Miwakeic i I Go ma a M S Co e Co cu e EEG/ M I a a ysis y mu iway a ia eas Squa es NeuroImage, (3 1 3-1 3 [191] S A o s G Sim so A M a e W e i eau A K iu A Ko e o a i a e M uo i ai e oo e A o e a I mo iemi S a io em o a ac i i y o a co ica e wo k o ocessi g isua mo io e ea e y MEG a M I J Neurophysiol, (5 5 5- 555 1999 [19 ] eis ei e M E e C eic meis e M iem i g E Mose E wa a eecke Mag e oe ce a og a y may e o im o e u c io a M I ai ma i g Eur J Neurosci, 9(5 1 7 -1 77 1997 [193] S Go a e A i o a ke G a G u a G a e e e a a Me e e e use o u c io a co s ai s o e eu oe ec omag e ic i e se o em A e a i es a ca ea s Int J Bioelectromag, 3(1 1 19
[19 ] C C is ma M u aus a o Simu a eous e ec oe ce a og a y a u c io a mag e ic eso a ce imagi g o ima y a seco a y soma ose so y co e i uma s a e e ec ica s imu a io Neurosci Lett, 333(1 9-73 [195] Ko e C imsky M Mo e as ei e a usc a Ga s a Co e a io o se so imo o ac i a io wi u c io a mag e ic eso a ce imagi g a mag e oe ce a og a y i esu gica u c io a imagi g a s a ia a a ysis Neurolmage, 1 (5 1 1 -1 1 [19 ] M Wag e a M uc s I eg a io o u c io a M I s uc u a M I EEG a MEG Int J Bioelectromag, 3(1 1 [197] K Ma iak a A a ga e Com i i g mag e oe ce a og a y a u c io a mag e ic eso a ce imagi g Int Rev Neurobiol, 1 1- 5 [19 ] o e M E McCou a C a i ig emis e e co o o isuos a ia a e io i e- isec io u gme s e a ua e wi ig - e si y e ec ica ma i g a sou ce a a ysis Neurolmage, 19(3 71 -7 3 [199] A M a e a M I Se e o Im o e oca i a io o co ica ac i i y y com i i g EEG a MEG wi M I co ica su ace eco s uc io A i ea a oac J Cog Neurosci, 5( 1 -17 1993 [ ] S Geo ge C Ai e C Mos e M Sc mi M a ke A Sc i C C Woo ewi e A Sa e s a W e i eau Ma i g u c io i e uma ai wi mag e oe ce a og a y a a omica mag e ic- eso a ce-imagi g a u c io a mag e ic- eso a ce-imagi g J Clin Neurophysiol, 1 (5 - 31 1995 [ 1] A K iu W e i eau a A M a e S a io em o a imagi g o uma ai ac i i y usi g u c io a M I co s ai e mag e oe ce a og a y a a Mo e Ca o simu a io s Proc Natl Acad Sci US A 95(15 9 5- 95 199 [ ] S A o s a G Sim so Geome ica i e e a io o M I-gui e MEG/EEG i e se es ima es Neurolmage, (1 3 3-33 [ 3] A M a e A K iu isc ewi e uck e W e i eau a E a g e y amic s a is ica a ame e ma i g com i i g M I a MEG o o uce ig eso u io imagi g o co ica ac i i y Neuron, 55- 7 [ ] a i o i C a i o i Ca ucci A ge o e C e -G a a G oma i M ossi i a Ci co i i ea i e se es ima io o co ica sou ces y usi g ig eso u io EEG a M I io s Int J Bioelectromag, 3(1 1 [ 5] M uc s M Wag e Ko e a Wisc ma i ea a o i ea cu e e si y eco s uc io s J Clin Neurophysiol, 1 (3 7- 95 1999 193
[ ] - a aye S ai e - o i e a Ga e o usio o simu a eous M I/EEG a a ase o e e ec o-me a o ic cou i g I Proc IEEE ISBI, ages - 7 A i g o i gi ia A [ 7] u i o- a e o E Ma í e -Mo es Me ie-Ga cía a A a és-Sosa A symme ica ayesia mo e o M I a EEG/MEG eu oimage usio Int J Bioelectromag, 3(1 1 [ ] u i o- a e o E Au e - a que a A a es-Sosa ayesia mo e a e agi g i EEG/MEG imagi g Neurolmage, 1( 13 -1319 [ 9] Me ie-Ga cía u i o- a e o E Ma í e -Moi es Koe ig a A a és-Sosa EEG imagi g ia MA wi M I e- e i e io mo e o a i i ies I Hum Brain Mapp, u a es u ga y u e [ 1 ] M Sa a i a S S aga a a eco s uc i g MEG sou ces wi u k ow co e a io s I Adv in Neu Info Proc Sys 16. MI ess [ 11] M Sc mi S Geo ge a C C Woo ayesia i e e ce a ie o e e ec omag e ic i e se o em Hum Brain Mapp, 7(3 195- 1 1999 [ 1 ] M Si g S Kim a S Kim Co e a io e wee O - M I a EEG sig a c a ges i es o se o isua s imu us eque cy i uma s Magn Reson Med, 9(1 1 -11 3 [ 13] Ma u i M M Mu ay A Meu i i a Mae e C M Mic e G a e e e a a Me e e a S Go a e A i o Me o s o e e mi i g eque cy- a egio - e e a e a io s i s e wee es ima e s a O es o ses i uma s J Neurophysiol, [ 1 ] S a a a o a g e C ecke M o mes W Mi e K Kai a a oi io Sca - eco e s ow EEG es o ses ge e a e i es o se o emo y amic c a ges i e uma ai Clin Neurophysiol, 11 (9 17 -175 3 [ 15] S C u A ea mu e a G o e MEG sou ce oca i a io usi g a M wi a is i u e ou u e ese a io IEEE Trans Biomed Eng, 5 ( 7 - 9 u e 3 [ 1 ] S C u a A ea mu e as o us su ec -i e e e mag e oe ce a og a ic sou ce oca i a io usi g a a i icia eu a e wo k Hum Brain Mapp, (1 1-3 5 [ 17] S ae ig e e Sc a ow e Sc eic A ec ma a C S e ma Sou e e e e e ce o au i o y e oke o e ia s simu a eous EEG eco i g a ow- oise M I Int J Psychophysiol, 7 (3 35- 1 19
[ 1 ] A oo u M K ame Ca ace o M Kaise C a ies au Ko e a M W i i g o em o a i e ac io s e wee co ica y ms Frontiers in Neuroscience, ( 1 5-15 [ 19] C -C C a g a C - i LIBSVM: a library for support vector machines, 1 So wa e a ai a e a //www csie u e u w/ c i / i s m [ ] S So e u g G ae sc C Sc ae e a Sc oe ko a ge sca e mu i e ke e ea i g Journal of Machine Learning Research, 7 1531-15 5 [ 1] E o e o I o s o e a i s i a i eas a g e eg essio Annals of Statistics, 3 7- 99 [ ] E Sc a i e e oos i g a oac o mac i e ea i g A o e iew I e iso M a se C o mes Ma ick a Yu e i o s Nonlinear Estimation and Classification. S i ge ew Yo k 3 [ 3] essoa a S a ma a eco i g ea - es o e ce io o ea om is i u e si g e- ia ai ac i a io Cerebral Cortex, 17 91-7 1 7 ISS 1 7-3 11 [ ] K iegesko e E o misa o So ge a Goe e I i i ua aces e ici is i c es o se a e s i uma a e io em o a co e Proceedings of the National Academy of Sciences of the USA, 1 - 5 7 ISS 1 91- 9 [ 5] - ay es a G ees eco i g me a s a es om ai ac i i y i uma s Nature Reviews Neuroscience, 7 5 3-53 ISS 1 71- 3 [ ] amsey S a so C a so Y a c e ko A o ack a C G ymou Si o ems o causa i e e ce om M I Su mi e [ 7] A o ack Y a c e ko a S a so eco i g e a ge-sca e s uc u e o ai u c io y c assi yi g me a s a es ac oss i i i ua s Psychological Science, I ess [ ] - ay es e ec i g ece io om eu oimagi g sig a s—a a a- i e e s ec i e Trends Cogn Sci, 1 ( 1 -7; au o e y 1 7- A ISS 13 - 13 [ 9] K E Si A oe s o W McG ego a C i e ec i g ece io e sco e a imi s Trends Cogn Sci, 1 ( -53 e ISS 13 - 13 [ 3 ] K E Si A oe s o W McG ego a C i es o se o ay es e e s mo e o ece io a ai ac i i y Trends in Cognitive Sciences, 1 ( 1 7-1 ISS 13 - 13 [ 31] C Mos e M ea y a S ewis EEG a MEG o wa so u io s o i e se me o s IEEE Trans Biomed Eng, (3 5- Ma 1999 195
[ 3 ] M A a ie A s u y o e e ec ic ie a e su ace o e ea Electroencephalogr Clin Neurophysiol, 3 -5 19 9 [ 33] G W uis Gi i g a M J. e e s A com a iso o i e e ume ica me o s o so i g e o wa o em i EEG a MEG Physiol Meas, 1 Su A A1-9 1993 [ 3 ] a g A as me o o com u e su ace o e ia s ge e a e y i o es wi i mu i aye a iso o ic s e es Phys Med Biol, 335-3 9 May 1995 [ 35] G o e e mag e ic ea ie eo em i e quasi-s a ic a o ima io a i s use o mag e oe ce a og a y o wa ca cu a io i ea is ic o ume co uc o s Phys. Med. Biol., 3 37-3 5 o 3 [ 3 ] G o e e mag e ic ea ie eo em i e quasi-s a ic a o ima io a i s use o MEG o wa ca cu a io i ea is ic o ume co uc o s Physics in Medicine and Biology, ( 3 37-5 [ 37] M ämä äi e a i J. I mo iemi K uu i a a 0. ou asmaa Mag e oe ce a og a y— eo y i s ume a io a a ica io s o o i asi e s u ies o e wo ki g uma ai Rev. Modern Physics, 5 ( 13-97 1993
[ 3 ] S ai e C Mos e a M ea y E ec omag e ic ai ma i g IEEE Sig Proc Mag, 1 ( 1 -3 o 1 [ 39] auk Kee i sim e a case o usi g c assica mi imum o m es ima io i e a a ysis o EEG a MEG a a Neurolmage, 1( 1 1 -1 1 [ ] S ai e a Ga e o A ayesia a oac o i o uci g a a omo- u c io a io s i e EEG/MEG i e se o em IEEE Trans Biomed Eng, (5 37 -3 5 May 1997 [ 1] G a e e e a a Me e e M M Mu ay C M Mic e Ma u i a S Go a e A i o E ec ica eu oimagi g ase o io ysica co s ai s Neurolmage, 1( 5 7-539 [ ] G Go u M ea a G Wa a Ge e a i e c oss- a i a io as a me o o c oosi g a goo i ge a ame e Technometrics, 1 15- 3 1979 [ 3] C a se A a ysis o isc e e i - ose o ems y mea s o e -cu e I SIAM Review, o ume 3 ages 5 1-5 Socie y o I us ia a A ie Ma ema ics i a e ia A USA 199 [ ] C i i s M D. ugg a K is o Sys ema ic egu a i a io o i ea i e se so u io s o e EEG sou ce oca i a io o em Neurolmage, 17 (1 7-3 1 196
[ 5] Ma ou C i i s W e y M ugg a K J. is o MEG sou ce oca i a io u e mu i e co s ai s a e e e ayesia amewo k Neurolmage, 3 (3 753- 7 [ ] ooks G A ma S Mac eo a G M Ma a os I e se e ec oca iog a y y simu a eous im osi io o mu i e co s ai s IEEE Trans Biomed Eng, (1 3-1 1999 [ 7] C awso a a so Solving Least Squares Problems. Se ies i Au oma ic Com u a io e ice- a E g ewoo C i s 7 3 USA 197 IS -13- 5 5- [ ] e s ea y a M Si g A e a ua io o me o s o eu omag e ic image eco s uc io IEEE Trans Biomed Eng, 3 (9 713-7 3 19 7 [ 9] i Wi e S A o s S M S u e eam W e i eau a M S ama ai e Assessi g a im o i g e s a ia accu acy i MEG sou ce oca i a io y e -weig e mi imum- o m es ima es Neurolmage, 31 (1 1 -71 [ 5 ] C i i s M ugg a K is o A a omica y i o me asis u c io s o EEG sou ce oca i a io com i i g u c io a a a a omica co s ai s NeuroImage, 1 (3 1 7 - 95 [ 51] G ackus a Gi e e eso i g owe o g oss Ea a a Geophys J R Astron Soc, 1 1 9- 5 19 [ 5 ] G a e e e a a Me e e a S Go a e A i o A c i ica a a ysis o i ea i e se so u io s o e eu oe ec omag e ic i e se o em IEEE Trans Biomed Eng, ages - 199 [ 53] Wa g S Wi iamso a Kau ma Mag e ic sou ce images e e mi e y a ea - ie a a ysis e u ique mi imum- o m eas -squa es es ima io IEEE Trans Biomed Eng, 39(7 5- 75 199 [ 5 ] ascua -Ma qui C M Mic e a e ma ow eso u io e ec omag e ic omog a y A ew me o o oca i i g e ec ica ac i i y o e ai Int J Psychophysiol, 1 9- 5 199 [ 55] a ee W a o ge e M Yuc ma a A Su uki oca i a io o ai e ec ica ac i i y ia i ea y co s ai e mi imum a ia ce s a ia i e i g IEEE Trans Biomed Eng, (9 7- 1997 [ 5 ] S E o i so a a u c io a eu oimagi g y sy e ica e u e mag e ome y (SAM I Yos imo o M Ko a i S Ku iki Ka i e a akasa o e i o s Recent Advances in Biomagnetism, ages 3 -3 5 Se ai a a 1999 o oku U i ess 197
[257] J. Vrba and S. E. Robinson. Differences between Synthetic Aperture Magnetometry (SAM) and linear beamformers. In J. Nenonen, R. Ilmoniemi, and T. Katila, editors, 12th Int Conf Biomagnet, 12th International Conference on Biomagnetism, Helsinki, Finland, Aug. 2000. Biomag2000. ISBN 951-22-5402-6.
[258] L. J. Waldorp, H. M. Huizenga, A. Nehorai, R. P. Grasman, and P. C. Molenaar. Model selection in spatio-temporal electromagnetic source analysis. IEEE Trans Biomed Eng, 52(3):414-20, 2005.
[259] J. Malmivuo and R. Plonsey. Bioelectromagnetism—Principles and Applications of Bioelectric and Biomagnetic Fields. Oxford University Press, New York, 1995, 1995.
[260] R. M. Leahy, S. Baillet, and J. C. Mosher. Integrated matlab toolbox dedicated to magnetoencephalography (MEG) and electroencephalography (EEG) data visualization and processing, 2004.
[261] NeuroFEM. Finite element software for fast computation of the forward solution in EEG/MEG source localisation, 2005. Max Planck Institute for Human Cognitive and Brain Sciences.
[262] BioPSE. Problem solving environment for modeling, simulation, and visualization of bioelectric fields, 2002. Scientific Computing and Imaging Institute (SCI).
[263] SCIRun. SCIRun: a scientific computing problem solving environment, 2002. Scientific Computing and Imaging Institute (SCI).
[264] E Poupon. Parcellisation systématique du cerveau en volumes d'intérêt. Le cas des structures profondes. Phd thesis, INSA Lyon, Lyon, France, Dec. 1999.
[265] FreeSurfer. FreeSurfer, 2004. CorTechs and the Athinoula A. Martinos Center for Biomedical Imaging.
[266] D. Van Essen. Surface reconstruction by filtering and intensity transformations, 2004.
[267] D. Shattuck and R. Leahy. BrainSuite: an automated cortical surface identification tool. Med Image Anal, 6(2):129-142, 2002. [268] D. Weber. EEG and MRI Matlab toolbox, 2004.
[269] J. E. Moran. MEG tools for Matlab software, 2005.
[270] A. Delorme and S. Makeig. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods, 134(1):9-21, 2004.
[271] F. A Nielsen. Bibliography of segmentation in neuroimaging, 2001.