XXVI SIMPÓSIO BRASILEIRO DE TELECOMUNICAÇÕES - SBrT’08, 02-05 DE SETEMBRO DE 2008, RIO DE JANEIRO, RJ
Relevance Vector Machine Applied to EEG Signals Classification Sandro Chagas, Marcio Eisencraft, Clodoaldo Ap. M. Lima
Abstract —The electroencephalogram (EEG) is a complex gives better frequency resolution. Also AR method has an and aperiodic time series, which is a sum over a very large advantage over FFT that, it needs shorter duration data number of neuronal membrane potentials. Despite the rapid records than FFT [20]. advances of neuroimaging techniques, EEG recording con- In the late 1890s, a powerful method was proposed to tinues playing an important role in both the diagnosis of perform time-scale analysis of signals: the wavelet trans- neurological diseases and understanding of the psychological process. In order to extract relevant information of brain forms (WT). This method provides a unified framework electrical activity, a variety of computerized-analysis me- for different techniques that have been developed for var- thods have been used. In this paper, we propose the use of a ious applications. Since the WT is appropriate for analysis recently developed machine-leaning technique – relevance of non-stationary signals and this represents a major ad- vector machine (RVM) – for EEG signals classification. vantage over spectral analysis, it is well suited to locating RVM is based on Bayesian estimation theory, which has as transient events. Wavelet’s feature extraction and repre- distinctive feature the fact that it can yield a sparse decision sentation properties can be used to analyze various tran- function defined only by a very small number of so-called re- sient events in biological signals. In [1] was presented an levance vectors. From the experimental results, we can see overview of the discrete wavelet transform (DWT) devel- that estimation and classification based on RVM perform well in EEG signals classification problem compared with oped for recognizing and quantifying spikes, sharp waves traditional approach support vector machine (SVM), which and spike-waves. Through, wavelet decomposition of the indicates that this classification method is valid and has EEG records, transient features are accurately captured promising application. and localized in both time and frequency context. Various other techniques from the theory of signal ana- Keywords - Electroencephalogram, Support Vector Ma- lyses have been used to obtain representations and extract chine, Relevance Vector Machine, Bayesian estimation the features of interest for classification purposes. Neural theory, classification problem networks and statistical pattern recognition methods have been applied to EEG analysis. In [11] was used the raw EEG data as an input to a neural network while in [19]
I. INTRODUCTION was used the features proposed by Gotman with an adap- The human brain is a complex system, and exhibits rich tive structure neural network, but his results show a poor spatiotemporal dynamics. Among the techniques for in- false detection rate. In [10] a recurrent neural network vestigating human brain dynamics, electroencephalogra- combined with wavelet pre-processing was proposed to phy (EEG) provides a non-invasive and direct measure of predict the onset of epileptic seizures both on scalp and cortical activity with temporal resolution in milliseconds. intracranial recordings only one-channel of electroence- EEG is a record of the electrical potentials generated by phalogram. the cerebral cortical neurons. Early on, EEG analysis was However, most of the techniques used to train the neu- restricted to visual inspection of EEG records. Since there ral network classifiers are based on the idea of minimi- is no definite criterion evaluated by the experts, visual zing the training error, which is usually called empirical analysis of EEG signals is insufficient. Routine clinical risk. As a result, limited amounts of training data and over diagnosis requires the analysis of EEG signals. Therefore, high training accuracy often lead to over training instead some automation and computer techniques have been of good classification performance. In addition, its classi- used for this aim [5]. Since the early days of automatic fication accuracy is also sensitive to the dimension of the EEG processing, representations based on a Fourier trans- training set. form have been most commonly applied. This approach is On the other hand, the Support Vector Machine (SVM) based on earlier observations that the EEG spectrum con- approach is based on the minimization of the structural tains some characteristic waveforms that fall primarily risk [18], which asserts that the generalization error is de- within four primary components. Such methods have limited by the sum of the training error and a parcel that proved beneficial for various EEG characterizations, but depends on the Vapnik-Chervonenkis dimension. By mi- fast Fourier transform (FFT), suffers from large noise nimizing this summation, high generalization perfor- sensitivity. Parametric power spectrum estimation me- mance may be obtained. Besides, the number of free pa- thods such as AR, reduces the spectral loss problems and rameters in SVM does not explicitly depend upon the in- put dimensionality of the problem at hand. Another im- Sandro Chagas, Marcio Eisencraft, Clodoaldo Ap. M. Lima, Escola de portant feature of the support vector learning approach is Engenharia, Universidade Presbiteriana Mackenzie, São Paulo, Brasil, that the underlying optimization problems are inherently E-mails: [email protected], [email protected], convex and have no local minima, which comes as the re- [email protected]. XXVI SIMPÓSIO BRASILEIRO DE TELECOMUNICAÇÕES - SBrT’08, 02-05 DE SETEMBRO DE 2008, RIO DE JANEIRO, RJ sult of applying Mercer's conditions on the characteriza- physik/eegdata.html [2]. The complete data set consists of tion of kernels [13]. five sets (denoted A-E) each containing 100 single- Although the SVM classification provides successful channel EEG segments. These segments were selected results, a number of significant and practical disadvantag- and cut out from continuous multi-channel EEG record- es are identified as follows [15][16]: ings after visual inspection for artifacts, e.g., due to mus- • Although SVMs are relatively sparse, the number of cle activity or eye movements. Sets A and B consisted of support vector (SVs) typically grows linearly with the segments taken from surface EEG recordings that were size of the training set and therefore, SVMs make carried out on five healthy volunteers using a standardized unnecessarily liberal of basic functions. electrode placement scheme (Figure 1). • Predictions are not probabilistic, and therefore, SVM is not suitable for classification tasks in which post- erior probabilities of class membership are necessary. • In SVM, it is required to estimate the error/margin trade tradeoff parameter C, which generally entails a cross-validation procedure which can be a waste of data as well as computation. • In SVM, the kernel function must satisfy Mercer’ condition, hence, it must be a continuous symmetric kernel of a positive integral operator. The Relevance Vector Machine (RVM) has been intro- duced by [15][16] as a Bayesian treatment alternative to the SVM that does not suffer from the aforementioned li- Figure 1: The 10-20 international system of electrode mitations. The RVM is a statistical learning method and it placement c images of normal and abnormal cases. is a probabilistic sparse kernel model identical in func- tional form to the SVM. It represents a new approach to Volunteers were relaxed in an awake-state with eyes pattern classification that has recently attracted a great open (A) and eyes closed (B), respectively. Sets C, D, and deal of interest in the machine learning community. RVM E originated from EEG archive of pre-surgical diagnosis. can be seen as a new way to train polynomial, neural net- EEGs from five patients were selected, all of whom had work, or Radial Basic Functions classifiers. It operates on achieved complete seizure control after resection of one the induction principle of structural risk minimization, of the hippocampal formations, which was therefore cor- which minimizes an upper bound on the generalization er- rectly diagnosed to be the epileptogenic zone. Segments ror. Because in this approach no probability density is es- in set D were recorded from within the epileptogenic timated, it becomes highly insensitive to the curse of di- zone, and those in set C from the hippocampal formation mensionality. In many problems RVM classifiers have of the opposite hemisphere of the brain. While sets C and been shown to perform much better than other non-linear D contained only activity measured during seizure free in- classifiers such as artificial neural networks. However, tervals, set E only contained seizure activity. Here seg- their application to EEG signals classification problem ments were selected from all recording sites exhibiting ic- has been very limited. tal activity. All EEG signals were recorded with the same In this paper, we propose the use of RVM for EEG sig- 128-channel amplifier system, using an average common nals classification. Furthermore, a comparative study in reference. The data were digitized at 173.61 samples per terms of performance and complexity (number of relev- second using 12 bit resolution. Bandpass filter settings ance vectors versus number support vectors) is realized. were 0.53-40 Hz (12dB/oct). In this work, like in [14], we As in traditional pattern-recognition systems, our ap- used two sets (A and E). proach consists of two main modules: a feature extractor based on DWT that generate a feature vector from the B. Data preprocessing EEG signals and feature classifier that output the class Wavelet transform is a spectral estimation technique in based on the features vector. which any general function can be expressed as an infinite In the next section we describe the data analyzed and series of wavelets. The basic idea underlying wavelet the techniques used to preprocess it. In Section 3 we analysis consists of expressing a signal as a linear combi- present the classifiers implement using the SVM and nation of a particular set of functions (WT), obtained by RVM; in Section 4 we show the results of the classifica- shifting and dilating one single function called a mother tions and the accuracy rate; and then in Section 5 we con- wavelet. The decomposition of the signal leads to a set of clude the study. coefficients called wavelet coefficients. Therefore the signal can be reconstructed as a linear combination of the II. DATA ANALYSIS wavelet functions weighted by the wavelet coefficients. In order to obtain an exact reconstruction of the signal, ade-
A. Data selection quate number of coefficients must be computed. The key In this work, we have used the EEG data publicly avai- feature of wavelets is the time-frequency localization. It lable at ( http://www.meb.uni-bonn.de/epileptologie/science/ means that most of the energy of the wavelet is restricted XXVI SIMPÓSIO BRASILEIRO DE TELECOMUNICAÇÕES - SBrT’08, 02-05 DE SETEMBRO DE 2008, RIO DE JANEIRO, RJ to a finite time interval. Frequency localization means that The generalized optimal separating hyperplane is de- the Fourier transform is band limited. When compared to termined by the vector w, that minimizes the functional: N FFT, the advantage of time-frequency localization is that 1 T min J (w,b,e) = 2 (w w) + C∑ ξt (1) wavelet analysis varies the time-frequency aspect ratio, w,b,e t=1 producing good frequency localization at low frequencies (where C is a given value) subject to the constraints T (long time windows), and good time localization at high yt [w ϕ(xt ) + b] ≥ 1− ξt , t = ,1 L, N (2) frequencies (short time windows). This produces a seg- Then, the resulting quadratic programming (QP) prob- mentation, or tiling of the time-frequency plane that is ap- lem may be written as: propriate for most physical signals, especially those of a N 1 N N α T max J( ) =max ∑αi − ∑∑αiα j yi yjφ(xi ) φ(xj ) transient nature. The wavelet technique applied to the α α (3) i=1 2 i=1j = 1 EEG signal will reveal features related to the transient na- N subject to ∑ α i yi = 0 and 0≤ α i ≤ C , for i = ,1 , N , for ture of the signal which are not obvious by the Fourier i=1 L T transform [6]. i = ,1 L, N . To obtain φ(xi ) φ(x j ) in QP problem we do Selection of suitable wavelet and the number of de- not need to calculate φ(x ) and φ(x ) explicitly. Instead, composition levels is very important in analysis of signals i j using the DWT. The number of decomposition levels is for some φ, we can design a kernel K(.,.) such that T chosen based on the dominant frequency components of K(x i , x j ) = φ(x i ) φ(x j ) . the signal. The levels are chosen such that those parts of Then, the expression of QP problem becomes: N N N the signal that correlate well with the frequencies neces- α 1 max J ( ) = max ∑α i − ∑∑α iα j yi y j K(xi ,x j ) α α (4) sary for classification of the signal are retained in the i=1 2 i=1j = 1 wavelet coefficients. In the present study, since the EEG For the training samples along the decision boundary, signals do not have any useful frequency components the corresponding α ' s are greater than zero, as ascer- above 30 Hz, the number of decomposition levels was i tained by the Kuhn-Tucker Theorem [13].These samples chosen to be 5. Thus, the EEG signals were decomposed are known as support vectors. The number of support vec- into details D1-D5 and one final approximation, A5 [14]. tors is generally much smaller than N, being proportional We have used a Daubechies order-4 wavelet (db4), its to the generalization error of the classifier [18]. A test smoothing feature made it more appropriated to detect m changes of EEG signal [14]. vector x ∈ℜ is then assigned to a given class with re- spect to the expression
C. Feature Extraction T N f (x) = sign (w φ(x) + b) = sign (∑ α i yi K(x, xi ) + b) In order to reduce the dimensionality of the extracted i=1 features vector [4][14][17][12], statistics over the set of B. Relevance Vector Machine the wavelet coefficients were used to generate the input to The RVM introduces a priori over the model weights the SVM: governed by a set of hyper-parameters, in a probabilistic Statistics over wavelet coefficients obtained: framework. One hyperparameter is associated with each • Average of wavelet coefficients in each sub-band weight, and the most probable values are iteratively esti- (W_Avg). mated from the training data. The most compelling fea-
• Standard deviation of wavelet coefficients in each ture of the RVM is that it typically utilizes significantly sub-band (W_Std). fewer kernel function compared to the SVM, while pro-
• Maximum of wavelet coefficients in each sub-band viding a similar performance. (W_Max). For two-class classification, any target can be classified into two class such that . A Bernoulli distribution
III. VARIANTS OF SUPPORT VECTOR MACHINES can be adopted for t) {0,1}in the probabilistic framework In this section we revise the use of SVM and RVM in because only two classes ( | (0 and 1) are possible. The lo- classification problems. gistic sigmoid link function is applied to y(x) to link random σand(y systematic= 1 (1 + expcomponents, (− A. Standard SVM and generalize the linear model. Following the definition N m Let the training set be {( xi , yi )} i=1 , with input xi ∈ ℜ of the Bernoulli distribution, the likelihood is written as and yi ∈ {±1}. The SVM first accomplishes a mapping φ: m n (5) ℜ → ℜ . Usually, n is much higher than m in such a way ( | = { ( ; } (1 − { ( ; } that the input vector is mapped into a high-dimensional for the targets ∈ {0,1}. space. When data are linearly separable, the SVM builds a The likelihood is complementary by a prior over the pa- n T hyperplane in ℜ w φ(x)+b in which the boundary be- rameter (weights) in the form of tween positive and negative samples is maximized. It can be shown that w, for this optimal hyperplane, may be de- (6) ( | fined as a linear combination of φ (x ) , that is = exp − i 2 2 N where shows the hyperparameter in- w= α yφ(x ). = ( , , ⋯ , ∑i=1 i i i troduced to control the strength of the priori over its asso- XXVI SIMPÓSIO BRASILEIRO DE TELECOMUNICAÇÕES - SBrT’08, 02-05 DE SETEMBRO DE 2008, RIO DE JANEIRO, RJ
ciated weight. Hence, the prior is Gaussian, but condi- fore not employed in the provided results. Note that tioned on determines the variance in the case of the RBF and ERBF For a certain . value, the posterior weight distribution kernel. conditioned on the data can be obtained using Bayes’ rule, • Linear Kernel i. e, , = . ( | ( | (7) • Polynomial Kernel ( | , = where p(t| w) is likelihood of t,( p|( w | is the prior density , = . of w, and p(t| is referred to as evidence. • RBF Kernel The weight cannot be analytically obtained, and there- fore, a Laplacian approximation procedure is used [8]. − , = − 1) Since p(w|t, ααα) is linearly proportional to 2 , it is possible to aim to find the maximum ( of | × • ERBF Kernel
( , − 1 log { ( | ( | } = [ + (1 − log (1 − ] , = − 2 2 for the most probable weight , with A. Configuration of the experiments = { ( ; } and being composed of the cur- In the experiments accomplished, as mentioned before, = ( , , ⋯ , rent values of ααα. This is a penalized logistic log- we have assessed the performance of the RVM and SVM likelihood function and requires iterative maximization. models with regard to the variation of kernel parameter, The iteratively reweighed least-squares algorithm can be keeping the value of the regularization parameter C con- used to find [16]. stant in 100. This value was achieved after some prelimi- 2) The logistic log-likelihood [15][16] function can be differen- nary experiments and agrees with the fact that SVM mod- tiated twice to obtain the Hessian in the form of els with low values of C tend in general to achieve better (8) performance than those with high values of this parame- ∇ ∇ ( | , | = −(∅ ∅ + where is a diagonal matrix with ter. Although we know that there are several rules-of- = ( , , ⋯ , and is the design thumb to select the values kernel parameter [3], for RBF = { ( ; }[1 − { ( ; }] ∅ matrix with and . This result and ERBF kernel we have opted to set the values of σ as ∅ = ( , ∅ = 1 is then negated and inverted to give the covariance Σ, as 4,2,1,5.0 and for polynomial kernel we have opted to set shown as follows, for a Gaussian approximation to the the values of d as 1, 2, 3, 4. For each of the four values in posterior over weights centered at this range, a 10-fold cross-validation was performed to : (9) better gauge the average performance of the models. To In this way, the classification∑ = (∅ ∅ + problem is locally linearized define whether a sample will be a support vector or relev- around in a effective way with -6 ance vector was used a threshold equal to 10 , i. e, ΣΣΣ = ∅ . > 10 These equations =are ∅ basically + equivalent( − to the solution B. Results of a generalized least-square problem. After obtaining In Table 1, we provide the value(s) of the kernel para- the hyper-parameters are updated using meter and the correspondent number of support vector λ = , where is the ith posterior mean weight and (SV) for the SVM and number of relevance vector (RV) λ / is defined as λ Σ where Σ is the ith diagonal for the RVM, in terms of cross-validation, for each element of the covariance, = 1 − and can be regarded as a quadruple
4; produced a smaller number of SVs when compared to [3] Cherkassky, V ; Ma, Y., “Practical selection of SVM number of RVs used in RVM. This can be justified due to parameters and noise estimation for SVM regres- the very low value of the threshold used to define SVs sion.” Neural Networks, vol. 17, pp. 113-126, 2004. and RVs. For other kernel parameters value and features [4] Güler; I. ; Übeyli, E. D. “A modified mixture of ex- vector the number of the SVs was much bigger than the perts network structure for ECG beats classification number of RVs. with diverse features.” Engineering Applications of From these results, it is possible to conclude that the Artificial Intelligence, vol. 18, pp. 845-856, 2005. choice of the kernel parameter value and kernel function [5] Guler, I., Kiymik, M. K., Akin, M., Alkan, A. “AR was not so much an important factor to be considered to spectral analysis of EEG signals by using maximum distinguish between the overall best error rates exhibited likelihood estimation.” Computers in Biology and by the machines. Medicine, vol. 31, pp. 441–450, 2001. When comparing the results obtained using features [6] Kara; S.; Okandan, M. “Atrial fibrillation classifica- vector and EEG data without pre-processing, one ob- tion with artificial neural networks.” Pattern Recogni- served that the performance is quite similar. This indi- tion, vol. 40, pp. 2967 – 2973, 2007. cates that the selection of parameters to the kernel is more [7] Lima, C. A. M., Villanueva, W. J. P., dos Santos, E. important than the technique used for the extraction of P. and Von Zuben, F. J. “A multistage ensemble of features. support vector machine variants.” In: Procs. of the 5th International Conference on Recent Advances in V. CONCLUDING REMARKS Soft Computing, Nottingham, pp. 670-675, 2004.
In this paper, we have presented a preliminary analysis [8] Mackay, D. J. C. “The evidence framework applied study contrasting as the performance exhibited by SVM to classification networks.” Neural Computation, vol. 4, no. 5, pp. 720-736, 1992 and RVM classifiers as number of the SVs and RVs with respect to the calibration of the kernel parameter value [9] Nabney, I. T. “Efficient training of RBF networks for and vector feature applied to EEG signals classification classification.” In Proc. 9th ICANN, 1999, vol. 1, pp. 210-215. problem. Such study is interesting as it can provide hints on how these machines are affected by the hyper- [10] Petrosian, A., Prokhorov, D., Homan, R., Dashei, R., parameter tuning process and feature vector extracted in Wunsch, D. “Recurrent neural network based predic- the EEG signal classification problem. tion of epileptic seizures in intraand extracranial EEG.” Neurocomputing, vol. 30, pp. 201–218, 2000. Experimental results show that RVM is superior to SVM in terms of the number of kernel functions that [11] Pradhan, N., Sadasivan, P. K., Arunodaya, G. R. needs to be used in the classification phase. Therefore, “Detection of seizure activity in EEG by an artificial RVM is preferable SVM in applications that require low neural network: A preliminary study.” Computers complexity and, possibility, real-time classification with a and Biomedical Research, vol. 29, pp. 303–313, 1996. priori training. As ongoing work, we are currently extending the scope [12] Revett, K. Jahankhani, P.; Kodogiannis, V. "EEG of investigation by considering multiclass problems, other Signal Classification Using Wavelet Feature Extrac- kernel functions (and parameters), other types of vector tion and Neural Networks." IEEE John Vincent Ata- machines — such as the Proximal SVMs, the Lagrangian nasoff 2006 International Symposium on Volume, pp. 120 – 124, 2006. SVMs [7], as well as (and most importantly) the conjoint influence of the hyper-parameters. In the future, we plan [13] Shawe-Taylor, J, Cristianini, N. "An Introduction to Support Vector Machines", Cambridge. Press, 2000. to investigate how the combination of models coming from different types of vector machines, each configured [14] Subasi, “A. EEG signal classification using wavelet with the same values of the control parameters, can im- feature extraction and a mixture of expert model.” prove the levels of performance, in terms of accuracy and Expert Systems with Applications, vol. 32, pp.1084- 1093, 2007. generalization, from that achieved by each vector machine type alone. [15] Tipping, M. “Sparse bayesian learning and the relev- ance vector machine.” Journal of Machine Learning Research, pp. 211-244, 2001. VI. REFERENCES [16] Tipping, M. “The Relevance Vector Machine.” Ad- [1] Adeli H, Zhou Z, Dadmehr N. “Analysis of EEG re- vances in Neural Information Proceeding systems 12. cords in an epileptic patient using wavelet trans- Cambreidge, Mass: MIT Press, 2000. form.” Journal Neuroscience Methods, vol. 123, no. [17] Übeyli, E. D. “Wavelet/mixture of experts network 1, pp. 69–87, 2003. structure for EEG signals classification.” Expert Sys- [2] Andrzejak, R. G.; Lehnertz, K.; Mormann, F.; Rieke, tems with Applications, vol. 34, pp. 1954-1962, C.; David, P. Elger, C. E. “Indications of nonlinear 2008. deterministic and finite-dimensional structures in [18] Vapnik, V. N. "The Nature of Statistical Learning time series of brain electrical activity: Dependence on Theory", Springer Verlag, 1995. recording region and brain state.” Phys. Rev. E, vol. [19] Weng, W., Khorasani, K. “An adaptive structure neu- 64, 061907, 2001. ral network with application to EEG automatic sei- XXVI SIMPÓSIO BRASILEIRO DE TELECOMUNICAÇÕES - SBrT’08, 02-05 DE SETEMBRO DE 2008, RIO DE JANEIRO, RJ
zure detection.” Neural Networks, vol. 9, pp. 1223– tions on Biomedical Engineering, vol. 45, pp. 673– 1240, 1996. 685, 1998. [20] Zoubir, M., Boashash, B. ‘Seizure detection of new- born EEG using a model approach”. IEEE Transac-
Table 1 - Comparative analysis: For each quadruple