Expert Systems With Applications 79 (2017) 44–52

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Linear classifier design under in Linear Discriminant Analysis

∗ Kojo Sarfo Gyamfi , James Brusey, Andrew Hunt, Elena Gaura

Faculty of Engineering and Computing, Coventry University, Coventry, CV1 5FB, United Kingdom

a r t i c l e i n f o a b s t r a c t

Article history: Under normality and assumptions, Linear Discriminant Analysis (LDA) is known to be Received 12 November 2016 optimal in terms of minimising the Bayes error for binary classification. In the heteroscedastic case, LDA

Revised 23 February 2017 is not guaranteed to minimise this error. Assuming heteroscedasticity, we derive a linear classifier, the Accepted 24 February 2017 Gaussian Linear Discriminant (GLD), that directly minimises the Bayes error for binary classification. In Available online 24 February 2017 addition, we also propose a local neighbourhood search (LNS) algorithm to obtain a more robust clas- Keywords: sifier if the data is known to have a non-normal distribution. We evaluate the proposed classifiers on LDA two artificial and ten real-world datasets that cut across a wide of application areas including Heteroscedasticity handwriting recognition, medical diagnosis and remote sensing, and then compare our algorithm against Bayes error existing LDA approaches and other linear classifiers. The GLD is shown to outperform the original LDA

Linear classifier procedure in terms of the classification accuracy under heteroscedasticity. While it compares favourably with other existing heteroscedastic LDA approaches, the GLD requires as much as 60 times lower train- ing time on some datasets. Our comparison with the support vector machine (SVM) also shows that, the GLD, together with the LNS, requires as much as 150 times lower training time to achieve an equivalent classification accuracy on some of the datasets. Thus, our algorithms can provide a cheap and reliable option for classification in a lot of expert systems. ©2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license. ( http://creativecommons.org/licenses/by-nc-nd/4.0/ )

C 1. Introduction Let K be the number of classes, k be the kth class, x be a fea- D ture vector and k be training samples belonging to the kth class In many applications one encounters the need to classify a (k ∈ { 1 , 2 , . . . , K} ) . The MAP decision rule for the classification task given object under one of a number of distinct groups or classes is then to choose the most likely class of x, C ∗( x ) given as: based on a set of features known as the feature vector. A typical ∗ C ( x ) = arg max p(C | x ) , k ∈ { 1 , 2 , . . . , K} (1) example is the task of classifying a machine part under one of a C k k number of health states. Other applications that involve classifica- tion include face detection, object recognition, medical diagnosis, We assume for the that there are only K = 2 classes, credit card fraud prediction and machine fault diagnosis. i.e. binary classification (we consider multi-class classification in a A common treatment of such classification problems is to later section). Then, using Bayes’ rule, the two posterior probabili- model the conditional density functions of the feature vector ( Ng & ties can be expressed as: Jordan, 2002 ). Then, the most likely class to which a feature vector p( x| C ) × p(C ) belongs can be chosen as the class that maximises the a posteriori (C | ) = k k , ∈ { , } p k x k 1 2 (2) probability of the feature vector. This is known as the maximum a p ( x ) posteriori (MAP) decision rule. It is often the case that the prior probabilities p(C 1 ) and p(C 2 ) are known, or else they may be estimable from the relative fre- quencies of D 1 and D 2 in D where D = D 1 ∪ D 2 . Let these priors be given by π and π respectively for class C and C . Then, the ∗ 1 2 1 2 Corresponding author. likelihood ratio defined as: E-mail addresses: gyamfi[email protected] (K.S. Gyamfi), [email protected] (J. Brusey), [email protected] (A. Hunt), p( x| C 1 ) [email protected] (E. Gaura). λ( x ) = (3) p( x| C 2 ) http://dx.doi.org/10.1016/j.eswa.2017.02.039 0957-4174/© 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license. ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ) K.S. Gyamfi et al. / Expert Systems With Applications 79 (2017) 44–52 45 is compared against a threshold defined as τ = π2 /π1 so that one As a algorithm, LDA is performed either decides on class C 1 if λ( x ) ≥ τ and class C 2 otherwise. for dimensionality reduction (usually followed by classification) Linear Discriminant Analysis (LDA) proceeds from here with ( Barber, 2012 , Chapter 16; Buturovic, 1994; Duin & Loog, 2004; two basic assumptions ( Izenman, 2009 , Chapter 8): Sengur, 2008 ), or directly for the purpose of statistical classification ( Fukunaga, 2013 , Chapter 4; Izenman, 2009; Mika et al., 1999 ). LDA 1. The conditional probabilities p( x| C ) and p( x| C ) have multi- 1 2 has been applied to several problems such as medical diagnosis variate normal distributions. e.g. Coomans, Jonckheer, Massart, Broeckaert, and Blockx (1978) ; 2. The two classes have equal covariance matrices, an assumption Polat, Güne s¸ , and Arslan (20 08) ; Sengur (20 08) ; Sharma and Pali- known as homoscedasticity. wal (2008) , face and object recognition e.g. Chen, Liao, Ko, Lin, and ,  D , Let x¯ 1 1 be the and covariance matrix of 1 and x¯ 2 Yu (2000); Liu, Chen, Tan, and Zhang (2007); Song, Zhang, Wang,  D ∈ 2 be the mean and covariance of 2 respectively. Then, for k Liu, and Tao (2007); Yu and Yang (2001) and credit card fraud pre- {1, 2}, diction e.g. Mahmoudi and Duman (2015) . The widespread use of   LDA in these areas is not because the datasets necessarily satisfy 1 1 T −1 ( | C ) =  − ( − )  ( − ) p x k exp x x¯ k k x x¯ k the normality and homoscedasticity assumptions, but mainly due ( π ) d ( ) 2 2 det k to the robustness of LDA against noise, being a linear model ( Mika (4) et al., 1999 ). Since the linear Support Vector Machine (SVM) can be quite expensive to train, especially for large values of K or n where d is the dimensionality of X , which is the feature space of ( n = n + n ), LDA is often relied upon ( Hariharan, Malik, & Ra- x. Given the above definitions of the conditional probabilities, one 1 2 manan, 2012 ). may obtain a log-likelihood ratio given as: Yet, practical implementation of LDA is not without problems. ln λ( x ) Of note is the small sample size (SSS) problem that LDA faces with    1 det 2 1 T −1 T −1 high-dimensional data and much smaller training data (Lu, Platan- = ln + ( x −x¯ )  ( x −x¯ ) −( x −x¯ )  ( x − x¯ ) 2 2 2 1 1 1 iotis, & Venetsanopoulos, 2003; Sharma & Paliwal, 2015 ). When d 2 det 1 2  n , the scatter matrix  is not invertible, as it is not full-rank. (5) x Since the decision rule as given by (7) requires the computation τ C which is then compared against ln so that 1 is chosen if of the inverse of x , the singularity of x makes the solution in- λ( ) ≥ τ, C ln x ln and 2 otherwise. Thus, the decision rule for clas- feasible. In works by, for example, Liu et al. (2007) ; Paliwal and C sifying a vector x under class 1 can be rewritten as: Sharma (2012) , this problem is overcome by taking the Moore– 2 Penrose pseudo-inverse of the scatter matrix, rather than the ordi- − − τ det  ( − ) T  1( − ) − ( − ) T  1( − ) ≥ 1 x x¯ 2 2 x x¯ 2 x x¯ 1 1 x x¯ 1 ln nary matrix inverse. Sharma and Paliwal (2008) use a gradient de- det 2 scent approach where one starts from an initial solution of w and (6) moves in the negative direction of the gradient of Fisher’s criterion In general, this result is a quadratic discriminant. However, a linear (8) . This method avoids the computation of an inverse altogether. classifier is often desired for the following reasons: Another approach to solving the SSS problem involves adding a scalar multiple of the identity matrix to the scatter matrix to make 1. A linear classifier is robust against noise since it tends not to the resulting matrix non-singular, a method known as regularised overfit ( Mika, Ratsch, Weston, Scholkopf, & Mullers, 1999 ). discriminant analysis ( Friedman, 1989; Lu et al., 2003 ). 2. A linear classifier has relatively shorter training and testing However, for a given dataset that does not satisfy the ho- times ( Yuan, Ho, & Lin, 2012 ). moscedasticity or normality assumption, one would expect that 3. Many linear classifiers allow for a transformation of the original modifications to the original LDA procedure accounting for these feature space into a higher dimensional feature space using the violations would yield an improved performance. One such mod- kernel trick for better classification in the case of a non-linear ification, in the case of a non-normal distribution, is the mixture decision boundary ( Bishop, 2006 , Chapter 6). discriminant analysis ( Hastie & Tibshirani, 1996; Ju, Kolaczyk, & By calling on the assumption of homoscedasticity, i.e. 1 = Gopal, 2003; McLachlan, 2004 ) in which a non-normal distribu- 2 = x , the original quadratic discriminant given by (6) for clas- tion is modelled as a mixture of Gaussians. However, the param- sifying a given vector x decomposes into the following linear deci- eters of the mixture components or even the number of mixture sion rule: components, are usually not known a priori. Other non-parametric

C 1 approaches to LDA that remove the normality assumption involve − 1 − − T  1( − )  τ + ( T  1 − T  1 ) x x x¯ 1 x¯ 2 ln x¯ 1 x x¯ 1 x¯ 2 x x¯ 2 (7) using local neighbourhood structures ( Cai, He, Zhou, Han, & Bao, C 2 2 2007; Fukunaga & Mantock, 1983; Li, Lin, & Tang, 2009 ) to con- −  1( − ) τ + struct a similarity matrix instead of the scatter matrix x used in Here, x x¯ 1 x¯ 2 is a vector of weights denoted by w and ln − − 1 ( T  1 − T  1 ) LDA. However, these approaches aim at linear dimensionality re- 2 x¯ 1 x x¯ 1 x¯ 2 x x¯ 2 is a threshold denoted by w0 . This linear classifier is also known as Fishers Linear Discriminant. If only the duction, rather than linear classification. Another modification, in D D , weight vector w is required for dimensionality reduction, w may the case of a non-linear decision boundary between 1 and 2 is be obtained by maximising Fishers criterion ( Fisher, 1936 ), given the Kernel Fisher Discriminant (KFD) ( Mika et al., 1999; Polat et al., by: 2008; Zhao, Sun, Yu, Liu, & Ye, 2009 ). KFD maps the original fea- X Y ture space into some other space (usually higher dimensional) w T ( x¯ − x¯ )( x¯ − x¯ ) T w S = 1 2 1 2 (8) via the kernel trick ( Mika et al., 1999 ). While the main utility of T  w x w the kernel is to guarantee linear separability in the transformed  =  +  D where x n1 1 n2 2 and n1 , n2 are the cardinalities of 1 and space, the kernel may also be employed to transform non-normal

D 2 respectively. data into one that is near-normal. LDA is the optimal Bayes’ classifier for binary classification if Our proposed method differs from the above approaches in that the normality and homoscedasticity assumptions hold ( Hamsici & we primarily consider violation of the homoscedasticity assump- Martinez, 2008 ) ( Izenman, 2009 , Chapter 8). It demands only the tion, and do not address the SSS problem. We seek to provide a computation of the dot product between w and x, which is a rela- linear approximation to the quadratic boundary given by (6) un- tively computationally inexpensive operation. der heteroscedasticity without any kernel transformation; we note Download English Version: https://daneshyari.com/en/article/4943493

Download Persian Version:

https://daneshyari.com/article/4943493

Daneshyari.com