IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1 Semi-Supervised Support Vector Machines with Tangent Space Intrinsic Manifold Regularization
Shiliang Sun and Xijiong Xie
Abstract—Semi-supervised learning has been an active research topic in machine learning and data mining. One main reason is that labeling examples is expensive and time-consuming while there are large numbers of unlabeled examples available in many practical problems. So far, Laplacian regularization has been widely used in semi-supervised learning. In this paper, we propose a new regularization method called tangent space intrinsic manifold regularization. It is intrinsic to data manifold and favors linear functions on the manifold. Fundamental elements involved in the formulation of the regularization are local tangent space representations which are estimated by local principal component analysis, and the connections which relate adjacent tangent spaces. Simultaneously, we explore its application to semi-supervised classification and propose two new learning algorithms called tangent space intrinsic manifold regularized support vector machines (TiSVMs) and tangent space intrinsic manifold regularized twin support vector machines (TiTSVMs). They effectively integrate the tangent space intrinsic manifold regularization consideration. The optimization of TiSVMs can be solved by a standard quadratic programming while the optimization of TiTSVMs can be solved by a pair of standard quadratic programmings. Experimental results on semi-supervised classification problems show the effectiveness of the proposed semi-supervised learning algorithms.
Index Terms—Support vector machine, twin support vector machine, semi-supervised classification, manifold learning, tangent space intrinsic manifold regularization ✦
1 INTRODUCTION classification. Typical multi-view classification meth- EMI-SUPERVISED classification, which estimates a de- ods include co-training [6], SVM-2K [7], co-Laplacian S cision function from few labeled examples and a SVMs [8], manifold co-regularization [9], multi-view large quantity of unlabeled examples, is an active re- Laplacian SVMs [10], sparse multi-view SVMs [11] and search topic. Its prevalence is mainly motivated by the multi-view Laplacian TSVMs [12]. Although the regular- need to reduce the expensive or time-consuming label ization method presented in this paper can be applied acquisition process. Evidence shows that, provided that to multi-view semi-supervised classification after some the unlabeled data which are inexpensive to collect appropriate manipulation, in this paper we only focus are properly exploited, people can obtain a superior on the single-view classification problem. performance over the counterpart supervised learning The principle of regularization has its root in mathe- approaches with few labeled examples. For a compre- matics to solve ill-posed problems [13], and is widely hensive survey of semi-supervised learning methods, used in statistics and machine learning [3], [14], refer to [1] and [2]. [15]. Many well-known algorithms, e.g., SVMs [16], Current semi-supervised classification methods can be TSVMs [17], ridge regression and lasso [18], can be divided into two categories, which are called single- interpreted as instantiations of the idea of regularization. view and multi-view algorithms, respectively. Their dif- A close parallel to regularization is the capacity control ference lies in the number of feature sets used to train of function classes [19]. Both regularization and the classifiers. If more than one feature set is adopted to capacity control can alleviate the ill-posed and over- learn classifiers, the algorithm would be called a multi- fitting problems of learning algorithms. Moreover, from view semi-supervised learning algorithm. The Lapla- the point of view of Bayesian learning, the solution to cian support vector machines (LapSVMs) [3], [4] and a regularization problem corresponds to the maximum Laplacian twin support vector machines (LapTSVMs) [5], a posterior (MAP) estimate for a parameter of interest. which can be regarded as two applications of Lapla- The regularization term plays the role of the prior dis- cian eigenmaps to semi-supervised learning, are rep- tribution on the parameter in the Bayesian model [20]. resentative algorithms for single-view semi-supervised In many real applications, data lying in a high- dimensional space can be assumed to be intrinsically Manuscript received 6 Dec. 2014; revised 7 May and 18 Jul. 2015; accepted of low dimensionality. That is, data can be well char- 23 Jul. 2015. This work was supported by the National Natural Science Foundation of China under Project 61370175 and the Science and Technology acterized by far fewer parameters or degrees of freedom Commission of Shanghai Municipality under Grant 14DZ2260800. than the actual ambient representation. This setting is Shiliang Sun and Xijiong Xie are with the Shanghai Key Laboratory of usually referred to as manifold learning, and the dis- Multidimensional Information Processing, Department of Computer Science and Technology, East China Normal University, 500 Dongchuan Road, tribution of data is regarded to live on or near a low- Shanghai 200241, P.R. China. E-mail: [email protected]. dimensional manifold. The validity of manifold learning IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2 has already been testified by some recent developments, followed by Section 7 which discusses further refine- e.g., the work in [21], [22] and [23]. Laplacian regu- ments about the proposed methods and other possible larization is an important manifold learning method, applications. Concluding remarks are given in Section 8. which is used to exploit the geometry of the probabil- ity distribution by assuming that its support has the 2 METHODOLOGY OF THE TANGENT SPACE geometric structure of a Riemannian manifold. Graph- INTRINSIC MANIFOLD REGULARIZATION based learning methods often use this regularization to We are interested in estimating a function f(x) defined obtain an approximation to the underlying manifold. on M⊂ Rd, where M is a smooth manifold on Rd. We In particular, Laplacian regularization has been widely assume that f(x) can be well approximated by a linear used in semi-supervised learning to effectively combine function with respect to the manifold M. Let m be the labeled examples and unlabeled examples. For exam- dimensionality of M. At each point z ∈M, f(x) can be ples, LapSVMs and LapTSVMs are two representative ⊤ represented as a linear function f(x) ≈ bz + wz uz(x)+ semi-supervised classification methods with Laplacian 2 o(kx − zk ) locally around z, where uz(x)= Tz(x − z) is regularization. Laplacian regularized least squares are an m-dimensional vector representing x in the tangent regarded as a representative semi-supervised regression space around z, and Tz is an m × d matrix that projects x method with Laplacian regularization [3]. The optimiza- around z to a representation in the tangent space of M tion of such algorithms is built on a representer theorem at z. Note that in this paper the basis for Tz is computed that provides a basis for many algorithms for unsuper- using local principal component analysis (PCA) for its vised, semi-supervised and fully supervised learning. simplicity and wide applicability. In particular, the point In this paper, we propose a new regularization method z and its neighbors are sent over to the regular PCA called tangent space intrinsic manifold regularization procedure [25] and the top m eigenvectors of the d × d to approximate a manifold more subtly. Through this covariance matrix are returned back as rows of matrix regularization we can learn a linear function f(x) on the m Tz. The weight vector wz ∈ R is an m-dimensional manifold. The new regularization has potentials to be vector, and it is also the manifold-derivative of f(x) at z applied to a variety of statistical and machine learning with respect to the uz(·) representation on the manifold, problems. In later descriptions, Laplacian regularization which we write as ∇T f(x)|x=z = wz. is in fact only a part of the tangent space intrinsic Mathematically, a linear function with respect to the manifold regularization. manifold M, which is not necessarily a globally linear Part of this research has been reported in a short function in Rd, is a function that has constant manifold conference paper [24]. Compared to the previous work, derivative. However, this does not mean wz is a constant we have derived the formulation of the new regular- function of u due to the different coordinate systems ization in detail. While the previous work mainly con- when the “anchor point” z changes from one point to sidered data representation with the new regularization, another. This needs to be compensated using “connec- this paper considers a different task semi-supervised tions” that map a coordinate representation uz′ to uz for classification and exhibits the usefulness of the new any z′ near z. For points far apart, the connections are regularization method for this task. Two new learn- not of interest for our purpose, since coordinate systems ing machines TiSVMs and TiTSVMs are thus proposed. on a manifold usually change and representing distant TiSVMs integrate the common hinge loss for classifica- points with a single basis would thereby lead to a large tion, norm regularization, and the tangent space intrinsic bias. manifold regularization term, and lead to a quadratic To see how our approach works, we assume for programming problem, while TiTSVMs lead to a pair simplicity that Tz is an orthogonal matrix for all z: of quadratic programming problems. Semi-supervised ⊤ TzTz = I(m×m). This means that if x ∈ M is close to classification experiments with TiSVMs and TiTSVMs on ⊤ 2 z ∈ M, then x − z ≈ Tz Tz(x − z)+ O(kx − zk ). Now multiple datasets give encouraging results. consider x that is close to both z and z′. We can express The remainder of this paper is organized as follows. In f(x) both in the tangent space representation at z and Section 2, we introduce the methodology of the tangent z′, which gives space intrinsic manifold regularization. Then in Section 3 ⊤ ⊤ ′ 2 2 we generalize it based upon the popular weighted- bz + wz uz(x) ≈ bz′ + wz′ uz′ (x)+ O(kx − z k + kx − zk ). graph representation of manifolds from data, and re- ⊤ ⊤ That is, bz + wz uz(x) ≈ bz′ + w ′ uz′ (x) . This means that formulate the regularization term as a matrix quadratic z ⊤ ⊤ ′ form through which we can gain insights about the bz + wz Tz(x − z) ≈ bz′ + wz′ Tz′ (x − z ). regularization and draw connections with related reg- ⊤ ′ Setting x = z, we obtain bz ≈ bz′ + wz′ Tz′ (z − z ), and ularization methods. Moreover, this reformulation will ⊤ ′ ⊤ ⊤ ′ facilitate resolving semi-supervised classification tasks, bz′ + wz′ Tz′ (z − z )+ wz Tz(x − z) ≈ bz′ + wz′ Tz′ (x − z ). and the corresponding algorithms are given in Section 4 This implies that and Section 5, respectively. Experimental results which ⊤ ⊤ ⊤ ⊤ shows the effectiveness of the regularization method in wz Tz(x − z) ≈ wz′ Tz′ (x − z) ≈ wz′ Tz′ Tz Tz(x − z)+ semi-supervised classification are reported in Section 6, O(kx − z′k2 + kx − zk2). (1) IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 3
Since (1) holds for arbitrary x ∈ M close to z ∈ M, R({bz, wz}z∈Z ) in (2) using a symmetric weight matrix ⊤ ⊤ ⊤ ′ it follows that, wz ≈ wz′ Tz′ Tz + O(kz − z k) or wz ≈ W constructed from the above data collection Z. ⊤ ′ TzTz′ wz′ + O(kz − z k) . Entries in W characterize the closeness of different This means that if we expand at points z1,..., zk ∈ Z, points where the points are often called nodes in the and denote neighbors of zj as N (zj ), then the correct terminology of graphs. Usually there are two steps in- regularizer will be volved in constructing a weighted graph. The first step builds an adjacency graph by putting an edge between k R({bz, wz}z∈Z )= i=1 j∈N (zi) bzi − bzj two “close” points. People can choose to use parameter h R N ⊤ 2P P ⊤ 2 ǫ ∈ or parameter n ∈ to determine close points, −w T (zi − zj ) + γkw − T T w k . (2) zj zj zi zi zj zj 2 which means that two nodes would be connected if their i Euclidean distance is within ǫ or either node is among The function f(x) is approximated as follows the n nearest neighbors of the other as indicated by the ⊤ f(x)= bz(x) + wz(x)Tz(x) x − z(x) , (3) Euclidean distance. The second step calculates weights on the edges of the graph with a certain similarity where z(x) = arg minz∈Z kx − zk2. This is a very natural measure. For example, the heat-kernel method computes formulation for out-of-example extensions. weight Wij for two connected nodes i and j by
2 kx −x k − i j 2.1 Effect of Local PCA Wij = exp t (5) As we consider the setting of manifold learning and the where parameter t> 0, while for nodes not directly con- dimensionality of the tangent space is usually less than nected the weights would be zero [23]. One can also use ⊤ p the one of the outer space, the local PCA is used to the polynomial kernel to calculate weights Wij = (xi xj ) determine the local tangent space. If we don’t use local where parameter p ∈ N is the polynomial degree. The PCA and consider the original space, the above approach simplest approach for weight assignment is to adopt the is equivalent to considering a piecewise linear function, {0, 1} values where weights are 1 for connected edges ⊤ namely Tz = Tz = I. The corresponding expression of and 0 for others. the regularization becomes Therefore, the generalization of the tangent space in- trinsic manifold regularizer turns out to be k R({b , w } Z )= b − b z z z∈ i=1 j∈N (zi) zi zj k k h R({b , w } ∈Z )= Wij b − b P2 P z z z i=1 j=1 zi zj − ⊤ ( − ) + γk − k2 . h wzj zi zj wzi wzj 2 (4) 2P P − ⊤ T ( − ) + γk − T T ⊤ k2 . i wzj zj zi zj wzi zi zj wzj 2 (6) However, this will lead to higher dimensionality and i more parameters. The previous regularizer is a special case of (6) with The rationality of the proposed regularization prin- {0, 1} weights. The advantage of the new regularizer is ciple can be interpreted from the standpoint of effec- that the discrepancy of the neighborhood relationship is tive function learning. A globally linear function in the treated more logically. original ambient Euclidean space is often too simple, Now we reformulate the regularizer (6) into a canon- but instead a locally or piecewise linear function in this ical matrix quadratic form. The benefits include relat- space would be too complex, since it can have too many ing our regularization method with other regularization parameters to be estimated. The linear function with approaches and facilitating subsequent formulations on respect to the manifold as favored by the current regu- semi-supervised classification. In particular, we would larization method is a good trade-off between these two like to rewrite the regularizer as a quadratic form in situations. Basically it can be seen as a locally linear func- terms of a symmetric matrix S as follows, tion in the ambient space, but since the dimensionality ⊤ bz bz of the function weights is m rather than d, the number of 1 1 . . parameters to be learned is greatly reduced compared to . . the local linear function setting in the original Euclidean bzk bzk R({bz, wz}z∈Z )= S space. This reflects a good leverage between flexibility wz1 wz1 and manageability for effective function learning from . . . . data. w w zk zk ⊤ b b 3 GENERALIZATION AND REFORMULATION OF z1 z1 . . THE REGULARIZATION TERM . .
bzk S1 S2 bzk Relating data with a discrete weighted graph is a pop- = ⊤ , (7) wz1 S S3 wz1 ular choice, and there are indeed a large family of 2 . . graph-based statistical and machine learning methods. . . It also makes sense for us to generalize the regularizer w w zk zk IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 4
S1 S2 3.3 Term Three where ⊤ is a block matrix representation of S2 S3 matrix S and the size of S1 is k × k. In this paper, (i = 1,...,k) m k k we suppose that wzi is an -dimensional ⊤ Wij − 2(b − b )w T (zi − zj ) vector. Therefore, the size of S is (k + mk) × (k + mk). In zi zj zj zj order to fix S, we decompose (6) into four additive terms Xi=1 Xj=1 k k as follows and then examine their separate contributions ⊤ = Wij − 2(b − b )w Bji to the whole S zi zj zj Xi=1 Xj=1 k k 2 k k k k R({bz, wz}z∈Z )= Wij (bzi − bzj ) ⊤ ⊤ = 2 Wij (−bz w Bji)+2 Wij (bz w Bji) Xi=1 Xj=1 i zj j zj Xi=1 Xj=1 Xi=1 Xj=1 term one k k k k k k | {z } ⊤ ⊤ = Wij (−bzi Bjiwzj )+ Wij (−w Bjibzi ) ⊤ 2 zj + Wij wzj Tzj (zi − zj ) Xi=1 Xj=1 Xi=1 Xj=1 Xi=1 Xj=1 k k k k + b ( W B⊤)w + w⊤ ( W B )b term two zj ij ji zj zj ij ji zj |k k {z } Xj=1 Xi=1 Xj=1 Xi=1 ⊤ k k k + Wij − 2(bz − bz )w Tz (zi − zj ) i j zj j ⊤ ⊤ Xi=1 Xj=1 = Wij (−bzi Bjiwzj )+ bzi Fi wzi Xi=1 Xj=1 Xi=1 term three k k k | k k {z } ⊤ ⊤ + Wij (−w Bjibz )+ w Fibz , + γ W kw − T T ⊤w k2 . zj i zi i ij zi zi zj zj 2 Xi=1 Xj=1 Xi=1 Xi=1 Xj=1 k where we have defined vectors {Fj }j=1 with Fj = term four k i=1 Wij Bji. From this equation, we can give the formu- | {z } ⊤ lationP of S2, and then the S in (7), which is its transpose, 3.1 Term One 2 is ready to get. 1 2 ⊤ Suppose we define two block matrices S2 and S2 sized b b 2 k k z1 z1 k × mk each where the block size is 1 × m, and S2 is 2 . . Wij (bzi − bzj ) =2 . (D − W ) . , a block diagonal matrix. Set the (i, j)-th block (i, j = . . 1 ⊤ Xi=1 Xj=1 b b 1,...,k) of S2 to be −Wij Bji , and the (i,i)-th block (i = zk zk 2 ⊤ 1 2 1,...,k) of S2 to be Fi . Then, it is clear that S2 = S2 +S2 . where D is a diagonal weight matrix with Dii = k j=1 Wij . This means that term one contributes to S1 3.4 Term Four inP (7). Actually, we have S1 = 2(D − W ). ⊤ Denote matrix Tzi Tzj by Aij . Then, with γ omitted temporarily, 3.2 Term Two k k
Define vector Bji = Tzj (zi − zj ). Then, ⊤ 2 Wij kwzi − Tzi Tzj wzj k2 k k Xi=1 Xj=1 2 ⊤ k k Wij wz Tzj (zi − zj ) j 2 Xi=1 Xj=1 = Wij kwzi − Aij wzj k2 k k Xi=1 Xj=1 ⊤ 2 k k k k = Wij (wz Bji) j ⊤ ⊤ ⊤ Xi=1 Xj=1 = Wij wzi wzi + Wij wzj Aij Aij wzj k k Xi=1 Xj=1 Xi=1 Xj=1 ⊤ ⊤ k k = Wij wz BjiBjiwzj j ⊤ Xi=1 Xj=1 − 2Wij wzi Aij wzj k k k Xi=1 Xj=1 ⊤ ⊤ ⊤ k k k = wz Wij BjiBji wzj = wz Hiwzi , j i ⊤ ⊤ ⊤ Xj=1 Xi=1 Xi=1 = Diiwzi I(m×m)wzi + wzj ( Wij Aij Aij )wzj i j i k X=1 X=1 X=1 where we have defined matrices {Hj }j=1 with Hj = k k k ⊤ ⊤ i=1 Wij BjiBji . − 2Wij w Aij wz H zi j PNow suppose we define a block diagonal matrix S3 Xi=1 Xj=1 sized mk × mk with block size m × m. Set the (i,i)-th k k k H block (i = 1,...,k) of S to be Hi. Then the resultant ⊤ ⊤ 3 = wz (DiiI + Ci)wzi − wz (2Wij Aij )wzj , H i i S3 is the contribution of term two for S3 in (7). Xi=1 Xi=1 Xj=1 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 5
k where in the last line we have defined matrices {Cj }j=1 yi ∈{+1, −1} (i =1,...,ℓ). Only binary classification is k ⊤ considered in this paper. with Cj = i=1 Wij Aij Aij . 1 Now supposeP we define two block matrices S3 and We propose a new method named tangent space in- 2 S3 sized mk × mk each where the block size is m × m, trinsic manifold regularized SVMs (TiSVMs) for semi- 1 and S3 is a block diagonal matrix. Set the (i,i)-th block supervised learning, which attempts to solve the follow- 1 ing problem (i = 1,...,k) of S3 to be DiiI + Ci, and the (i, j)-th 2 block (i, j = 1,...,k) of S3 to be 2Wij Aij . Then the ℓ ℓ+u 1 2 1 contribution of term four for S3 would be γ(S − S ). 2 3 3 min (1 − yif(xi))+ + γ1 kwik2 {b , }ℓ+u ℓ Further considering the contribution of term two for S3, i wi i=1 Xi=1 Xi=1 (8) H 1 2 we finally have S3 = S3 + γ(S3 − S3 ). ℓ+u + γ2R({bi, wi}i=1 ),
where (1 − yif(xi))+ is the hinge loss [16], [26] defined 3.5 Connections with the Laplacian Regularization as From our reformulation, we can draw a connection 1 − yf(x), for 1 − yf(x) ≥ 0 (1 − yf(x)) = between our regularizer and the Laplacian regulariza- + 0, otherwise, tion [23]. The Laplacian regularizer, in our current ter- minology, can be expressed as γ1,γ2 ≥ 0 are regularization coefficients, and ℓ+u R({bi, wi}i=1 ) is the regularizer analogical to that k k 2 given in (6). Note that, for labeled examples, f(xi) is Wij (b − b ) zi zj equal to the corresponding bi. Xi=1 Xj=1 For classification purpose, the classifier outputs for the ⊤ bz1 bz1 unlabeled training examples are = 2 . (D − W ) . . . yi = sgn(bi), i = ℓ +1,...,ℓ + u, (9) b b zk zk ⊤ where sgn(·) is the sign function which outputs 1 if bz1 bz1 its inputs are nonnegative and −1 otherwise. For a test . . ℓ+u = 2 . L . , example x which is not in {xi}i=1 , the SVM classifier b b predicts its label by zk zk ⊤ where L = D−W is the Laplacian matrix. Obviously, the sgn bz(x) + wz(x)Tz(x) x − z(x) , (10) matrix S1 in the tangent space intrinsic manifold regu- where z(x) = arg min ℓ+u kx − zk2. larizer equals 2L. In this sense, we can say that our reg- z∈{xi}i=1 ularizer reflects the Laplacian regularization to a certain extent. However, this regularizer is more complicated as 4.2 Optimization via Quadratic Programming it intends to favor linear functions on the manifold, while We now show that the optimization of (8) can be im- the Laplacian regularization only requests the function plemented by a standard quadratic programming solver. values for connected nodes to be as close as possible. To begin with, using slack variables to replace the hinge loss, we rewrite (8) as
4 SEMI-SUPERVISED SVMS ℓ ℓ+u 1 2 In this section, we first introduce the model of TiSVMs. min ξi + γ1 kwik2 {b ,w }ℓ+u,{ξ }ℓ ℓ Then we transform the optimization problem to a i i i=1 i i=1 Xi=1 Xi=1 ℓ+u (11) quadratic programming problem. Finally, we give the + γ2R({bi, wi}i=1 ) computational complexity and algorithmic description of y b ≥ 1 − ξ , i =1,...,ℓ s.t. i i i TiSVMs. ξi ≥ 0, i =1,...,ℓ.
⊤ ⊤ ⊤ ⊤ ⊤ Define ξ = (ξ1,...,ξℓ) , and a = (ξ , b , w ) . Sup- 4.1 TiSVMs pose the first ℓ entries of b correspond to the ℓ labeled The SVM is a powerful tool for classification and regres- example x1,..., xℓ, respectively. We can reformulate (11) sion. It is based on structural risk minimization that min- as a standard quadratic program [27] imizes the upper bound of the generalization error [28]. 1 ⊤ ⊤ The standard SVMs solve a quadratic programming min a H1a + h1 a a 2 problem and output the hyperplane that maximizes the s.t. H2a ≤ h2 , (12) margin between two parallel hyperplanes. For the semi-supervised classification problem, sup- where H1 is a sparse matrix whose nonzero entries are br pose we have ℓ labeled examples and u unlabeled exam- only included in the bottom right sub-block H1 sized ℓ+u ⊤ 1 1 ples {xi}i=1 , and without loss of generalization the first (ℓ + u)(m + 1) × (ℓ + u)(m + 1), h1 = ( ℓ ,..., ℓ , 0,..., 0) ℓ examples correspond to the labeled ones with labels with the first ℓ entries being nonzero, H2 is a sparse IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 6
⊤ matrix with 2ℓ rows, h2 = (−1,..., −1, 0,..., 0) with the quadratic programming problems in turn front ℓ entries being −1 and the back ℓ entries being 0. ℓ2 ℓ1 br 1 The matrix H1 is given by 2(γ1Hd + γ2S) where Hd is a + 2 min ξi + γ1 kbi k2 diagonal matrix whose first ℓ + u diagonal elements are b+, + ℓ+u, ξ ℓ2 ℓ2 { i wi }i=1 { i}i=1 Xi=1 Xi=1 zero and the last m(ℓ + u) diagonal elements are 1, and + + ℓ+u (13) + γ2R({bi , wi }i=1 ) S is taken from (7) where we set k = ℓ + u and zi = xi + b ≥ 1 − ξi, i =1,...,ℓ2 (i =1,...,k). The nonzero entries of H2 can be specified s.t. ℓ1+i as follows. For row i (i ≤ ℓ), the i-th element is −1 and ξi ≥ 0, i =1,...,ℓ2, the (ℓ + i)-th element is −yi. For row ℓ + i (i ≤ ℓ), the and i −1 -th element is . ℓ1 ℓ2 TiSVMs solve a quadratic program and have the com- 1 − 2 min ηi + γ1 kbi k2 3 b−, − ℓ+u, η ℓ1 ℓ1 putational complexity of O([(ℓ + u)(m + 1)+ ℓ] ). For { i wi }i=1 { i}i=1 Xi=1 Xi=1 clarity, we explicitly state our tangent space intrinsic − − ℓ+u + γ2R({bi , wi }i=1 ) manifold regularized support vector machines algorithm − (14) b ≥ 1 − ηi, i =1,...,ℓ1 in Algorithm 1. ℓ2+i ηi ≥ 0, i =1,...,ℓ1 s.t. b−
+ where H1 is a sparse matrix whose nonzero entries are LapSVMs [3], [10], respectively. +br only included in the bottom right sub-block H1 sized In its most natural form, the optimization of soft- (ℓ+u)(m+1)×(ℓ+u)(m+1) h+⊤ = ( 1 ,..., 1 , 0,..., 0) margin SVMs for supervised classification is to find a , 1 ℓ2 ℓ2 + with the first ℓ2 entries being nonzero, H2 is a sparse function fs from a reproducing kernel Hilbert space +⊤ matrix with 2ℓ2 rows, and h2 = (−1,..., −1, 0,..., 0) (RKHS) H. Optimization in the original space is sub- with the front ℓ2 entries being −1 and the back ℓ2 entries sumed in the RKHS case with a linear kernel. Given +br + a set of ℓ labeled examples {(xi,yi)} (i = 1, ..., ℓ) with being 0. The matrix H1 is given by 2γ2S +Mℓ1 where + S is taken from (7) by setting k = ℓ+u and zi = xi (i = yi ∈{+1, −1}, the optimization problem is
1,...,k) and Mℓ1 is a diagonal matrix with the first ℓ1 ℓ 1 diagonal elements being 2γ1 and the rest 0. The nonzero 2 min (1 − yifs(xi))+ + γskfsk2, + fs∈H ℓ entries of H2 can be specified as follows. For row i (i ≤ Xi=1 ℓ ), both the i-th element and the (ℓ + ℓ + i)-th element 2 1 2 where γ ≥ 0 is a norm regularization parameter. are −1. For row ℓ + i (i ≤ ℓ ), the i-th element is −1. s 2 2 −1 Then we reformulate (14) as a standard quadratic Suppose examples belonging to classes 1 and are A B A program represented by matrices + and −, and the size of + and B− are (ℓ1 × d) and (ℓ2 × d), respectively. We define 1 −⊤ − − −⊤ − two matrices A, B and four vectors v1, v2, e1, e2, where min a H1 a + h1 a a− 2 e1 and e2 are vectors of ones of appropriate dimensions − − − s.t. H2 a ≤ h2 , (18) and − where H1 is a sparse matrix whose nonzero entries are w1 w2 −br A = (A+, e1), B = (B−, e2), v1 = , v2 = . (19) only included in the bottom right sub-block H1 sized b1 b2 (ℓ+u)(m+1)×(ℓ+u)(m+1) h−⊤ = ( 1 ,..., 1 , 0,..., 0) , 1 ℓ1 ℓ1 − TSVMs obtain two nonparallel hyperplanes with the first ℓ1 entries being nonzero, H2 is a −⊤ ⊤ ⊤ sparse matrix with 2ℓ1 + ℓ2 + ℓ1 rows, and h2 = w1 x + b1 =0 and w2 x + b2 =0 (20) (−1,..., −1, 0,..., 0,...,b+ ,...,b+ , −b+,..., −b+ ) ℓ1+1 ℓ1+ℓ2 1 ℓ1 around which the examples of the corresponding class with the front ℓ1 entries being −1, the back ℓ1 entries −br − get clustered. The classifier is given by solving the being 0. The matrix H1 is given by 2γ2S + Mℓ2 where S− is taken from (7) by setting k = ℓ + u and following quadratic programs separately (TSVM1) zi = xi (i = 1,...,k) and Mℓ2 is a diagonal matrix with the first ℓ2 diagonal elements being 2γ1 and the 1 ⊤ ⊤ − min (Av1) (Av1)+ c1e q , 2 1 rest 0. The nonzero entries of H2 can be specified as v1 q1 2 (21) follows. For row i (i ≤ ℓ1), both the i-th element and the s.t. − Bv1 + q1 e2, q1 0, (ℓ1 + ℓ2 + i)-th element are −1. For row ℓ1 + i (i ≤ ℓ1), the i-th element is −1. For row 2ℓ1 + i (i ≤ ℓ2), the (TSVM2) (ℓ + i) 1 2ℓ + ℓ + i i ≤ ℓ 1 -th element is . For row 1 2 ( 1), the 1 ⊤ ⊤ min (Bv2) (Bv2)+ c2e q (ℓ1 + ℓ2 + i)-th element is −1. , 1 2 v2 q2 2 (22) TiTSVMs solve a pair of quadratic programs and have 3 s.t. Av2 + q2 e1, q2 0, the computational complexity of O([(ℓ+u)(m+1)+ℓ1] + 3 [(ℓ + u)(m + 1) + ℓ2] ). For clarity, we explicitly state where c1, c2 are nonnegative parameters and q1, q2 our tangent space intrinsic manifold regularized twin are slack vectors of appropriate dimensions. The label support vector machines algorithm in Algorithm 2. of a new example x is determined by the minimum ⊤ of |x wr + br| (r = 1, 2) which are the perpendicular Algorithm 2 Tangent Space Intrinsic Manifold Regular- distances of x to the two hyperplanes given in (20). ized Twin Support Vector Machines (TiTSVMs) The intension of LapSVMs is for effective semi-
1: Input: ℓ labeled examples (ℓ1 positive examples and supervised learning under the local smoothness reg- ℓ2 negative examples), u unlabeled examples. ularization. The role of unlabeled data is to restrict + − + − + − + − the capacity of the considered function set through the 2: Obtain H1 , H1 , h1 , h1 , H2 , H2 , h2 , h2 . 3: Solve the quadratic programming (17) and (18) by Laplacian matrix L. Namely, desirable functions must be using cross-validation to choose parameters. smooth across all the training examples. Given ℓ labeled 4: Output: Predict the label of unlabeled training ex- and u unlabeled examples, LapSVMs solve the following amples according to (15); predict the label of a new problem in an RKHS example according to (16). ℓ 1 2 ⊤ min (1 − yifls(xi))+ + γ1kflsk2 + γ2fls Lfls, fls∈H ℓ Xi=1 5.3 Comparisons with Related Work where γ1 and γ2 are nonnegative regularization For the purpose of comparison, below we give the parameters as given before, and vector fls = ⊤ objective functions for SVMs [16], [28], TSVMs [17] and (fls(x1),...,fls(xℓ+u)) . IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 8
Fig. 1. Examples of digits 3 and 8 in the handwritten digit Fig. 2. Examples of face and non-face images in the face data set. detection data set.
SVMs, TSVMs and LapSVMs are important learning spaces are fixed to be 2-dimensional and 3-dimensional, machines making use of the “kernel trick” where a linear since for handwritten digits people usually obtain good classification function in a potentially high-dimensional embedding results with these dimensionalities. RKHS is learned, whose relationship with respect to With the identified regularization parameters, we re- the original inputs is usually nonlinear when nonlinear train TiSVMs and TiTSVMs, and evaluate performances kernel maps are adopted to generate the space. There on the test data and unlabeled training data, respec- are two main differences between them and our TiSVMs tively. The entire process is repeated over 10 random and TiTSVMs: i) They seek linear functions in probably divisions of the training and test sets, and the reported high-dimensional kernel spaces [19], while TiSVMs and performance is the averaged accuracy and the corre- TiTSVMs pursue linear functions in low-dimensional sponding standard deviation. Besides SVMs, LapSVMs tangent spaces of manifolds; ii) They only learn one sin- and LapTSVMs, parallel field regularization (PFR) [29] gle classification function, while TiSVMs and TiTSVMs is also adopted to compare with our method under learn many (bi, wi) pairs which are then dynamically the same setting, e.g., the same kernel is used, and selected to determine the label of some test example regularization parameters are selected with the same depending on the distances between this example and two-fold cross-validation method from the same range. the training examples. However, there is a common PFR, which is briefly mentioned in Section 7.4, is a recent property among all the considered learning machines. semi-supervised regression method which is used here That is, they are all able to learn nonlinear classifiers for classification. For SVMs the unlabeled training data with respect to the original feature representation. are neglected because they are only for semi-supervised learning. 6 EXPERIMENTS We evaluated the tangent space intrinsic manifold regu- 6.2 Face Detection larization for semi-supervised classification with multi- Face detection is a binary classification problem which ple real-world datasets. There is a parameter γ, which is intends to identify whether a picture is a human face or intrinsic in the regularization matrix S. This parameter not. In this experiment, 2000 face and non-face images leverages the two components in (6). Because there are from the MIT CBCL repository2 [31], [32] are used, where not very reasonable and necessary reasons to overweight half of them are faces and each image is a 19 × 19 gray one over the other, we treat them equally in S, namely picture. Figure 2 shows a number of examples. The same γ = 1. Other parameters in classifiers are selected experimental setting with the previous handwritten digit through cross-validation. For example, regularization classification is adopted, such as the percentage of train- parameters for TiSVMs and TiTSVMs, namely γ1 and ing data and the percentage of labeled data. −6 −4 −2 γ2, are selected from the set {10 , 10 , 10 , 1, 10, 100} through a two-fold cross-validation of labeled training 6.3 Speech Recognition examples on the training set. The Isolet dataset3 comes from the UCI Machine Learn- ing Repository. It is collected from 150 subjects speaking 6.1 Handwritten Digit Classification each letter of the alphabet twice. Hence, we have 52 This dataset comes from the UCI Machine Learning training examples from each speaker. Due to the lack of Repository. The data we use here include 2400 examples three examples, there are 7797 examples in total. These of digits 3 and 8 chosen from the MNIST digital images1, speakers are grouped into five sets of 30 speakers each. and half of the data are digit 3. Image sizes are 28 × 28. These groups are referred to as isolet1-isolet5. Each of Ten images are shown in Figure 1. The training set these datasets has 26 classes. The attribute information includes 80% of the data, and the remaining 20% serve as include spectral coefficients, contour features, sonorant the test set. For semi-supervised classification, 1% of the features, pre-sonorant features and post-sonorant fea- whole data set are randomly selected from the training tures. In Isolet dataset, we choose two classes (a, b) set to serve as labeled training data. for classification. So there are in total 480 examples. For the construction of adjacency graphs, 10 nearest The training set includes 80% of the data, and the neighbors are used. For edge weight assignment, we remaining 20% serve as the test set. For semi-supervised choose the polynomial kernel of degree 3, following [30] classification, 1/48 of the whole data set are randomly and [3] on similar classification tasks. The local tangent 2. http://cbcl.mit.edu/software-datasets/FaceData2.html 1. http://yann.lecun.com/exdb/mnist/ 3. http://archive.ics.uci.edu/ml/datasets/ IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 9
TABLE 1 Classification Accuracies (%) (with Standard Deviations) and Time (with Standard Deviations) of Different Methods on the Handwritten Digit Classification Data.
SVM LapSVM PFR(m=2) TiSVM(m=2) U 88.04 (1.78) 90.56 (1.74) 88.51 (6.64) 92.09 (1.58) T 87.98 (1.96) 90.40 (1.37) 88.56 (6.25) 92.40 (1.79) − − T 1 3.62×10 1(4.86×10 2) 3.03×102 (9.75) 6.05×103 (7.07×10) 6.70×103 (1.6×102) − − − − − − − − T 2 6.87×10 2 (2.11×10 2) 5.90×10 3 (1.40×10 3) 3.00×10 4 (4.80×10 4) 3.00×10 4 (4.80×10 4) −2 −4 −1 −2 −1 −2 −1 −2 T 3 1.55×10 (5.00×10 ) 1.55×10 (2.39×10 ) 1.57×10 (2.24×10 ) 1.55×10 (1.28×10 )
TiTSVM(m=2) PFR(m=3) TiSVM(m=3) TiTSVM(m=3) LapTSVM U 91.21 (1.99) 90.48 (2.63) 91.49 (1.88) 90.43 (2.52) 92.80 (1.71) T 91.48 (2.34) 90.73 (2.87) 91.90 (1.85) 90.71 (2.29) 90.31 (2.37) T 1 4.52×104 (8.26×102) 7.29×103 (2.13×102) 8.95×103 (2.08×102) 1.24×105 (9.31×103) 2.88×103 (1.96×102) −3 −4 −4 −4 −4 −3 −3 −1 T 2 2.00×10 (0.00) 4.00×10 (5.16×10 ) 4.00×10 (5.16×10 ) 4.50×10 (4.30×10 ) 1.03×10 (8.45×10 ) − − − − − − − T 3 2.22×10 1 (8.74×10 2) 1.89×10 1 (7.00×10 2) 1.85×10 1 (4.61×10 2) 1.96 (1.96) 2.59 (3.16×10 1)
TABLE 2 Classification Accuracies (%) (with Standard Deviations) and Time (with Standard Deviations) of Different Methods on the Face Detection Data.
SVM LapSVM PFR(m=2) TiSVM(m=2) U 76.42 (6.34) 79.42 (6.65) 80.25 (7.21) 85.42 (3.86) T 76.43 (6.12) 79.43 (5.65) 79.75 (7.76) 84.63 (3.64) − − T 1 3.58×10 1 (4.97×10 2) 1.87×102 (4.83) 1.97×103 (1.29×10) 2.27×103 (6.64×10) − − − − − − − − T 2 2.12×10 2 (2.70×10 3) 3.90×10 3 (8.75×10 4) 1.00×10 4 (3.16×10 4) 1.00×10 4 (3.16×10 4) −3 −3 −2 −3 −2 −3 −2 −3 T 3 5.50×10 (1.40×10 ) 7.70×10 (9.30×10 ) 7.21×10 (9.70×10 ) 6.40×10 (3.30×10 )
TiTSVM(m=2) PFR(m=3) TiSVM(m=3) TiTSVM(m=3) LapTSVM U 77.04 (10.78) 80.17 (7.77) 85.26 (4.11) 77.94 (7.79) 81.44 (2.77) T 77.22 (10.73) 79.95 (7.95) 84.58 (3.97) 78.70 (8.40) 81.12 (3.26) 4 2 3 3 2 4 3 3 T 1 2.68×10 (5.48×10 ) 2.89×10 (8.43×10) 4.17×10 (1.13×10 ) 6.69×10 (7.86×10 ) 1.77×10 (2.48×10) − − − − − − − − T 2 1.90×103 (3.16×10 4) 2.00×10 4 (4.22×10 4) 2.00×10 4 (4.22×10 4) 2.90×10 3 (3.20×10 3) 6.34 (3.00×10 1) − − − − − − − − − T 3 7.75×10 2 (1.82×10 2) 6.53×10 2 (1.19×10 2) 7.39×10 2 (9.70×10 3) 4.00×10 1 (5.24×10 1) 1.56 (5.13×10 2) selected from the training set to serve as labeled training randomly selected from the training set to serve as the examples. The same experimental setting is adopted as labeled training data. In this dataset, we use RBF kernel the previous experiments. and set the adjusting kernel parameter to be 100 which can perform well for LapSVMs. The other experimental 6.4 German Credit Data setting is adopted as the previous experiments. This German Credit dataset4 also comes from the UCI Machine Learning Repository and consists of 1000 exam- 6.6 Contraceptive Method Choice ples (300 positive examples and 700 negative examples). The Contraceptive Method Choice dataset6 contains 1473 The training set includes 80% of the data, and the examples containing three classes. We choose 629 pos- remaining 20% serve as the test set. For semi-supervised itive examples and 511 negative examples for binary classification, 1% of the whole data set are randomly classification. The training set includes 80% of the data, selected from the training set to serve as the labeled and the remaining 20% serve as the test set. For semi- training data. The same experimental setting is adopted supervised classification, 1/50 of the whole data set are as the previous experiments. randomly selected from the training set to serve as the labeled training data. In this dataset, we use RBF kernel 6.5 Australian and set the adjusting kernel parameter to be 100 which The Australian dataset5 contains 690 examples (307 can perform well for LapSVMs. The other experimental positive examples and 383 negative examples) and 14 setting is adopted as the previous experiments. attributes. The training set includes 80% of the data, and the remaining 20% serve as the test set. For semi- 6.7 Experimental Results supervised classification, 1/30 of the whole data set are Handwritten Digit Classification: The classification ac- 4. https://archive.ics.uci.edu/ml/datasets/ curacies and time of different methods on this dataset are Statlog+(German+Credit+Data) 5. http://archive.ics.uci.edu/ml/datasets/ 6. https://archive.ics.uci.edu/ml/datasets/ Statlog+%28Australian+Credit+Approval%29 Contraceptive+Method+Choice IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 10
TABLE 3 Classification Accuracies (%) (with Standard Deviations) and Time (with Standard Deviations) of Different Methods on the Isolet Data.
SVM LapSVM PFR(m=2) TiSVM(m=2) U 93.93 (3.30) 97.11 (0.95) 95.61 (8.01) 97.49 (0.63) T 94.16 (4.48) 97.08 (2.90) 95.42 (9.82) 98.23 (0.85) − − − T 1 4.20×10 2 (7.40×10 3) 4.54 (2.79×10 1) 5.22×102 (2.01×10) 3.54×102 (7.00) −2 −2 −4 −4 −4 −4 −4 −4 T 2 1.51×10 (3.16×10 ) 5.00×10 (5.30×10 ) 1.00×10 (3.16×10 ) 1.00×10 (3.16×10 ) −2 −4 −3 −3 −2 −2 −3 −3 T 3 1.30×10 (6.75×10 ) 8.60×10 (2.40×10 ) 1.98×10 (2.48×10 ) 9.90×10 (2.10×10 )
TiTSVM(m=2) PFR(m=3) TiSVM(m=3) TiTSVM(m=3) LapTSVM U 98.18 (0.92) 95.24 (8.53) 97.27 (0.47) 97.41 (1.78) 97.91 (0.85) T 98.65 (1.21) 95.42 (10.57) 97.81 (1.04) 98.23 (1.63) 98.33 (1.22) 3 2 2 2 3 T 1 1.08×10 (1.06×10 ) 3.97×10 (1.72×10) 4.25×10 (2.10×10) 1.67×10 (7.99×10) 9.30×10 (4.82) − − − − − − − − − − T 2 5.00×10 4 (5.27×10 4) 1.00×10 4 (4.25×10 4) 1.00×10 4 (3.164.25×10 4) 7.00×10 4 (4.83×10 4) 8.26×10 2 (1.47×10 2) − − − − − − − − − − T 3 1.26×10 2 (3.50×10 3) 1.25×10 2 (4.50×10 3) 1.04×10 2 (2.40×10 3) 1.15×10 2 (2.20×10 3) 2.48×10 2 (8.00×10 3)
TABLE 4 Classification Accuracies (%) (with Standard Deviations) and Time (with Standard Deviations) of Different Methods on the German Credit Data.
SVM LapSVM PFR(m=2) TiSVM(m=2) U 60.03 (7.63) 67.48 (4.22) 70.00 (0.00) 70.00 (0.00) T 59.50 (6.21) 66.55 (3.47) 70.00 (0.00) 70.00 (0.00) − − T 1 1.05×10 1 (1.35×10 1) 2.79×10 (1.86) 3.28×102 (8.32) 6.90×102 (2.58×102) −3 −3 −3 −4 −4 −4 −4 −4 T 2 1.10×10 (1.10×10 ) 1.10×10 (3.16×10 ) 1.00×10 (3.16×10 ) 1.00×10 (3.16×10 ) − − − − − − − − T 3 4.00×10 4 (6.99×10 4) 1.74×10 2 (3.10×10 3) 1.63×10 2 (1.01×10 2) 2.04×10 2 (1.96×10 2)
TiTSVM(m=2) PFR(m=3) TiSVM(m=3) TiTSVM(m=3) LapTSVM U 66.55 (0.92) 70.00 (0.00) 70.00 (0.00) 68.13 (3.79) 61.96 (7.44) T 67.55 (4.46) 70.00 (0.00) 70.00 (0.00) 67.10 (5.11) 62.30 (7.26) 3 2 2 3 2 2 T 1 4.42×10 (8.26×10) 4.14×10 (8.21) 4.92×10 (1.35×10) 9.18×10 (4.03×10 ) 3.40×10 (1.08×10) − − − − − − − − − T 2 1.00×10 3 (2.12×10 5) (8.21) 1.00×10 4 (3.16×10 4) 1.00×10 4 (3.16×10 4) 1.00×10 3 (0.00) 7.42×10 1 (6.71×10 2) − − − − − − − − − − T 3 1.14×10 2 (2.30×10 3) 1.16×10 2 (3.50×10 3) 1.06×10 2 (2.90×10 3) 2.5610 2 (4.83×10 2) 1.87×10 1 (2.87×10 2) shown in Table 1, where U and T represent the accuracy ods give better performance than the supervised SVMs. on the unlabeled training and test data, respectively, and Moreover, the proposed TiSVMs and TiTSVMs perform the best accuracies are indicated in bold. T 1, T 2 and much better than LapSVMs and PFR. LapTSVMs and T 3 (all in seconds) represent training time, test time on TiTSVMs perform a little better than TiSVMs. the unlabeled data and test time on the labeled data, German Credit Data: The classification accuracies and respectively. From this table, we see that semi-supervised time of different methods on this dataset are shown methods give better performance than the supervised in Table 4. Again, we see that semi-supervised meth- SVMs, which indicates the usefulness of unlabeled ex- ods give better performance than the supervised SVMs. amples. Moreover, the proposed TiSVMs and TiTSVMs Moreover, the proposed TiSVMs perform much better perform much better than LapSVMs and PFR. TiSVMs than TiTSVMs, LapSVMs, LapTSVMs and SVMs, and perform a little better than TiTSVMs. LapTSVMs perform perform the same as PFR. TiTSVMs perform a little better best on the classification of the unlabeled data while than LapSVMs on the classification of the labeled data TiSVMs performs best on the classification of the labeled and a little worse than LapSVMs on the classification of data. the unlabeled data. Face Detection: The classification results and time Australian: The classification accuracies and time of of TiSVMs and TiTSVMs with comparisons to other different methods on this dataset are shown in Ta- methods are given in Table 2, which show that our ble 5. We see that semi-supervised methods give better TiSVMs have got the best accuracies on both the test and performance than the supervised SVMs. Moreover, the unlabeled training data. However, the performance of proposed TiTSVMs perform much better than TiSVMs, our TiTSVMs is not good, just a little better than SVMs. LapSVMs, LapTSVMs and SVMs. We conjecture this is caused by the fact that TiTSVMs Contraceptive Method Choice: The classification ac- have more parameters to train and thus it is sometimes curacies and time of different methods on this dataset difficult to return a very good solution. are shown in Table 6. Semi-supervised methods give Speech Recognition: The classification accuracies and better performance than the supervised SVMs. TiTSVMs time of different methods on this dataset are shown in (m = 3) perform best on the classification of the unla- Table 3. This table shows that semi-supervised meth- beled data and TiSVMs (m = 2) perform best on the IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 11
TABLE 5 Classification Accuracies (%) (with Standard Deviations) and Time (with Standard Deviations) of Different Methods on the Australian Data.
SVM LapSVM PFR(m=2) TiSVM(m=2) U 56.25 (5.92) 60.44 (4.76) 61.70 (4.92) 61.02 (4.52) T 54.86 (7.60) 60.14 (6.12) 61.16 (5.10) 60.58 (3.17) − − − T 1 6.33×10 2 (9.98×10 2) 1.34×10 (9.56×10 1) 1.65×102 (6.80) 2.03×103 (7.16×102) −3 −3 −4 −4 −4 −4 −4 −4 T 2 1.50×10 (1.30×10 ) 5.00×10 (5.27×10 ) 1.00×10 (3.16×10 ) 1.00×10 (3.16×10 ) −4 −4 −2 −2 −3 −3 −3 −3 T 3 9.00×10 (5.68×10 ) 1.74×10 (3.86×10 ) 6.70×10 (1.70×10 ) 6.10×10 (1.90×10 )
TiTSVM(m=2) PFR(m=3) TiSVM(m=3) TiTSVM(m=3) LapTSVM U 64.64 (3.92) 62.20 (4.74) 62.10 (4.19) 62.36 (4.54) 59.96 (6.64) T 63.48 (6.49) 62.32 (6.74) 60.72 (4.19) 64.04 (6.94) 60.80 (11.84) 3 3 2 3 2 4 3 2 T 1 4.49×10 (1.14×10 ) 2.09×10 (7.67) 2.84×10 (7.47×10 ) 1.05×10 (2.16×10 ) 1.64×10 (1.68×10) − − − − − − − − − − T 2 8.00×10 4 (4.22×10 4) (8.21) 1.40×10 3 (4.10×10 3) 1.40×10 3 (4.10×10 3) 8.75×10 4 (3.53×10 4) 1.70×10 1 (1.85×10 2) − − − − − − − − − − T 3 7.70×10 3 (3.20×10 3) 7.90×10 3 (4.00×10 3) 7.90×10 3 (4.40×10 3) 1.92×10 2 (3.27×10 2) 4.08×10 2 (7.60×10 3)
TABLE 6 Classification Accuracies (%) (with Standard Deviations) and Time (with Standard Deviations) of Different Methods on the Contraceptive Method Choice Data.
SVM LapSVM PFR(m=2) TiSVM(m=2) U 55.81 (1.00) 56.20 (3.47) 56.53 (1.29) 56.71 (1.19) T 55.13 (2.08) 57.06 (3.93) 56.54 (3.48) 57.06 (3.79) − − − T 1 7.70×10 2 (1.02×10 2) 3.77×10 (1.37×10 1) 4.86×102 (2.66×10) 5.03×102 (9.39×10) −3 −4 −3 −4 −4 −4 −4 −4 T 2 1.10×10 (3.16×10 ) 1.40×10 (5.16×10 ) 3.00×10 (4.83×10 ) 3.00×10 (4.83×10 ) − − − − − − − − T 3 3.00×10 4 (4.83×10 4) 8.80×10 3 (6.32×10 4) 2.51×10 2 (1.09×10 2) 1.27×10 2 (2.20×10 3)
TiTSVM(m=2) PFR(m=3) TiSVM(m=3) TiTSVM(m=3) LapTSVM U 56.58 (1.74) 56.90 (1.41) 56.57 (1.18) 56.98 (0.02) 56.37 (6.05) T 56.14 (3.77) 56.89 (3.91) 56.71 (3.03) 56.62 (4.27) 56.93 (6.46) 3 2 2 2 3 2 2 T 1 4.70×10 (9.40×10 ) 6.41×10 (6.13) 5.60×10 (2.54×10) 8.68×10 (4.80×10 ) 4.45×10 (2.36×10) − − − − − − − − T 2 1.00×10 3 (0.00) 1.00×10 4 (3.16×10 4) 1.00×10 4 (3.16×10 4) 1.10×10 3 (3.16×10 4) 1.19 (6.95×10 2) −2 −3 −2 −3 −2 −2 −2 −2 −1 −2 T 3 1.84×10 (5.10×10 ) 2.30×10 (7.50×10 ) 1.89×10 (2.50×10 ) 2.02×10 (2.05×10 ) 3.10×10 (4.53×10 )
TABLE 7 Classifier Rank of All the Methods
method SVM LapSVM PFR(m=2) TiSVM(m=2) TiTSVM(m=2) PFR(m=3) TiSVM(m=3) TiTSVM(m=3) LapTSVM Average rank (U) 9.00 6.33 5.25 2.83 4.33 4.42 3.67 4.33 4.83 Average rank (T) 9.00 5.75 5.67 2.75 4.50 4.17 3.75 4.75 4.67 classification of the labeled data. 7.1 Out-of-Sample Extension Using Multiple Neigh- Table 7 lists the average rank of all the methods bors for their classification accuracies, from which we can In this paper, we only used one nearest neighbor from conclude our methods outperform other methods. From the training data to represent the out-of-sample exten- the perspective of time, as can be seen from Table 1∼6, sion for the learned functions, as given in (3). The they usually take more training time, less test time on performance would depend largely on the property of the unlabeled data and more test time on the labeled the selected neighboring point. data. However, in order to enhance the robustness, we can adopt a weighted average to represent the function f(x) for x out of the training set Z. Suppose we adopt n neighbors to carry out the out-of-example extension, and N (x) ⊂ Z includes the n neighbors of x. Then f(x) is 7 DISCUSSIONS computed as follows
1 ⊤ f(x)= Wxz bz + wz Tz x − z , Here we discuss some possible improvements and exten- Wxz z∈N (x) z∈NX(x) sions of the proposed tangent space intrinsic manifold P regularization and semi-supervised classification algo- where Wxz is the weight calculated with the same man- rithms, which can be very helpful to adapt to different ner as constructing the weighted graphs from Z. Here applications. n, which is not necessarily equal to the neighborhood IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 12 number used to construct the adjacency graphs, can regression with the squared loss be selected through some appropriate model selection ℓ ℓ+u procedure. (10) and (16) can be extended similarly. 1 2 2 min yi − f(xi) + γ1 kwik2 {b ,w }ℓ+u ℓ (23) i i i=1 Xi=1 Xi=1 ℓ+u 7.2 Reducing Anchor Points + γ2R({bi, wi}i=1 ). We treated each example from the training set as an The generalization to examples not in the training set is anchor point, where local PCA is used to calculate the analogical to (10) but without the sign function. tangent space. The number of parameters that should It can be easily shown that the optimization of (23) be estimated in our methods basically grows linearly is obtained by solving a sparse linear system, which is with respect to the number of anchor points. Therefore, much simpler than the quadratic programming for the in order to reduce the parameters to be estimated, one classification case. This is reminiscent of the works by possible approach is to reduce anchor points where [33] and [29], which are both on semi-supervised re- only “key” examples are kept as anchor points and the gression preferring linear functions on manifolds. These function values for the other examples are extended from works perform regression from the more complex per- those of the anchor points. This is a kind of research spectives of Hessian energy and parallel fields, respec- for data set sparsification. People can devise different tively. There are two main differences between (23) and methods to find the examples which they regard as their works: i) Their objective functions do not contain “key”. the second term of (23) and thus cannot regularize the ℓ+u The research of anchor point reduction is especially norms of {wi}i=1 ; ii) Their works focus on transductive useful whether training data are very limited or of large- learning while by solving (23) we can do both inductive scale. For limited data, the precision for parameter esti- and transductive learning. Comparing the performances mation can be improved as a result of parameter reduc- of these methods on regression is not the focus of this tion, while for large-scale data, anchor point reduction paper, which can be explored as interesting future work. can be promising to speed up the training process. For example, reducing anchor points can be applied to semi- 7.5 Entirely Supervised Learning supervised learning where the training data include a large number of unlabeled examples. The TiSVMs for semi-supervised classification can be extended to the entirely supervised learning scenario when all the training examples are labeled (a similar 7.3 Improving the Estimation of Tangent Spaces extension of TiTSVMs is also possible). In this case, the objective function could probably have the following For the manifold learning problems considered in this form paper, the estimation of bases for tangent spaces is an ℓ ℓ important step where local PCA with fixed neighbor- 1 2 min (1 − yif(xi))+ + γ1 kwik hood size was used. This is certainly not the optimal ℓ 2 {bi,wi}i=1 ℓ choice, since data could be non-uniformly sampled and Xi=1 Xi=1 manifolds can have a varying curvature. Notice that + γ2R({bi, wi}i∈I+ )+ γ3R({bi, wi}i∈I− ), the neighborhood size can determine the evolution of where I+ and I− are the index set for positive and neg- calculated tangent spaces along the manifold. If a small ative examples, respectively. The separation of positive neighborhood size is used, the tangent spaces would and negative examples reflects our intension to construct change more sharply when the manifold is not flat. a weight graph for each class [3], since we do not expect Moreover, noise can damage the manifold assumption that a positive example is among the neighbors for a as well to a certain extent. All these factors explain negative example or vice versa. The function values for the necessity for using different neighborhood sizes and the positive and negative examples would tend to be far more robust subspace estimation methods. apart, which is clearly an advantage. A more detailed In addition, data can exhibit different manifold dimen- study of the entirely supervised learning case is left for sions at different regions, especially for complex data. future work. Therefore, adaptively determining the dimensionality at different anchor points is also an important refinement concern of the current approach. 8 CONCLUSION In this paper, we have proposed a new regularization method called tangent space intrinsic manifold regular- 7.4 Semi-Supervised Regression ization, which favors linear functions on the manifold. The proposed tangent space intrinsic manifold regular- Local PCA is involved as an important step to estimate ization can also be applied to semi-supervised regression tangent spaces and compute the connections between problems. Using the same notations as in Section 4, we adjacent tangent spaces. Simultaneously, we proposed give the following objective function for semi-supervised two new methods for semi-supervised classification with IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 13 tangent space intrinsic manifold regularization. Experi- [22] J. Tenenbaum, V. de Silva, and J. Langford, ”A global geometric mental results on multiple datasets including compar- framework for nonlinear dimensionality reduction,” Science, vol. 290, no. 5500, pp. 2319–2323, 2000. isons with state-of-the-art algorithms have shown the [23] M. Belkin and P. Niyogi, ”Lapalcian eigenmaps for dimensionality effectiveness of the proposed methods. reduction and data representation,” Neural Computation, vol. 15, no. Future work directions include analyzing the general- 6, pp. 1373–1396, 2003. [24] S. Sun, ”Tangent space intrinsic manifold regularization for data ization error of TiSVMs and TiTSVMs, and applying the representation,” in Proceedings of the 1st IEEE China Summit and tangent space intrinsic manifold regularization to semi- International Conference on Signal and Information Processing, Beijing, supervised regression and supervised learning tasks. China, Jul. 2013, pp. 179–183. [25] I. T. Jolliffe, Principal Component Analysis. New York, NY: Springer- Verlag, 1986. [26] B. Sch¨olkopf and A. Smola, Learning with Kernels. Cambridge, MA: REFERENCES The MIT Press, 2001. [27] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, [1] O. Chapelle, B. Sch¨olkopf, and A. Zien, Semi-supervised Learning. England: Cambridge University Press, 2004. Cambridge, MA: The MIT Press, 2006. [28] J. Shawe-Taylor and S. Sun, ”A review of optimization method- [2] X. Zhu, ”Semi-supervised learning literature survey,” Department ologies in support vector machines,” Neurocomputing, vol. 74, no. of Computer Sciences, University of Wisconsin-Madison, Madison, 17, pp. 3609–3618, 2011. WI, Technical Report 1530, Jul. 2008. [29] B. Lin, C. Zhang, and X. He, ”Semi-supervised regression via par- [3] M. Belkin, P. Niyogi, and V. Sindhwani, ”Manifold regularization: allel field regularization,” Advances in Neural Information Processing A geometric framework for learning from labeled and unlabeled Systems, vol. 24, no. 1, pp. 433–441, 2012. examples,” Journal of Machine Learning Research, vol. 7, no. 11, pp. [30] B. Sch¨olkopf, C. Burges, and V. Vapnik, ”Extracting support data 2399–2434, 2006. for a given task,” in Proceedings of the 1st International Conference [4] R. Johnson and T. Zhang, ”On the effectiveness of Laplacian nor- on Knowledge Discovery and Data Mining, Montreal, Quebec, Aug. malization for graph semi-supervised learning,” Journal of Machine 1995, pp. 252–257. Learning Research, vol. 8, no. 7, pp. 1489–1517, 2007. [31] M. Alvira and R. Rifkin, ”An empirical comparison of SNoW and [5] Z. Qi, Y. Tian, and Y. Shi, ”Laplacian twin support vector machine SVMs for face detection,” Center for Biological and Computational for semi-supervised classification,” Neural Networks, vol. 35, no. 11, Learning, Massachussetts Institute of Technology, Cambridge, MA, pp. 46–53, 2012. Technical Report 193, Jan. 2001. [6] A. Blum and T. Mitchell, ”Combining labeled and unlabeled data [32] S. Sun, ”Ensembles of feature subspaces for object detection,” with co-training,” in Proceedings of the 11th Annual Conference on Lecture Notes in Computer Science, vol. 5552, no. 1, pp. 996–1004, Computational Learning Theory, Madison, WI, Jul. 1998, pp. 92–100. 2009. [7] J. Farquhar, D. Hardoon, H. Meng, J. Shawe-Taylor, and S. Szed- [33] K. Kim, F. Steinke, and M. Hein, ”Semi-supervised regression mak, ”Two view learning: SVM-2K, theory and practice,” Advances using Hessian energy with an application to semi-supervised di- in Neural Information Processing Systems, vol. 18, no. 1, pp. 355–362, mensionality reduction,” Advances in Neural Information Processing 2006. Systems, vol. 22, no. 1, pp. 979–987, 2010. [8] V. Sindhwani, P. Niyogi, and M. Belkin, ”A co-regularization approach to semi-supervised learning with multiple views,” in Proceedings of the Workshop on Learning with Multiple Views with the 22nd ICML, University of Bonn, Germany, Aug. 2005, pp. 1–6. [9] V. Sindhwani and D. Rosenberg, ”An RKHS for multi-view learn- ing and manifold co-regularization,” in Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, Jul. 2008, pp. 976–983. [10] S. Sun, ”Multi-view Laplacian support vector machines,” Lecture Shiliang Sun is a professor at the Department Notes in Computer Science, vol. 7121, no. 1, pp. 209–222, 2011. of Computer Science and Technology and the [11] S. Sun and J. Shawe-Taylor, ”Sparse semi-supervised learning head of the Pattern Recognition and Machine using conjugate functions,” Journal of Machine Learning Research, Learning Research Group, East China Normal vol. 11, no. 12, pp. 2423–2455, 2010. University. He received the Ph.D. degree from [12] X. Xie and S. Sun, ”Multi-view Laplacian twin support vector Tsinghua University, Beijing, China, in 2007. machines,” Applied Intelligence, vol. 41, no. 4, pp. 1059–1068, 2014. His research interests include kernel methods, [13] A. N. Tikhonov, ”Regularization of incorrectly posed problems,” learning theory, approximate inference, sequen- Soviet Mathematics Doklady, vol. 4, no. 6, pp. 1624–1627, 1963. tial modeling and their applications, etc. [14] T. Evgeniou, M. Pontil, and T. Poggio, ”Regularization networks and support vector machines,” Advances in Computational Mathe- matics, vol. 13, no. 1, pp. 1–50, 2000. [15] C. Hou, F. Nie, F. Wang, C. Zhang, and Y. Wu, ”Semisupervised learning using negative labels,” IEEE Transactions on Neural Net- works and Learning Systems, vol. 22, pp. 420–432, Mar. 2011. [16] V. N. Vapnik, Statistical Learning Theory. New York, NY: Wiley, 1998. [17] R. Jayadeva, S. Khemchandani, and Chandra, ”Twin support vector machines for pattern classification,” IEEE Transactions on Xijiong Xie is a Ph.D. student in the Pattern Pattern Analysis and Machine Intelligence, vol. 74, pp. 905–910, May Recognition and Machine Learning Research 2007. Group, Department of Computer Science and [18] R. Tibshirani, ”Regression shrinkage and selection via the lasso,” Technology, East China Normal University. Journal of the Royal Statistical Society: Series B Methodological, vol. 58, His research interests include machine learn- no. 1, pp. 267-288, 1996. ing, pattern recognition, etc. [19] J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge, England: Cambridge University Press, 2004. [20] D. Rosenberg, Semi-supervised Learning with Multiple Views. Ph.D. Thesis, Department of Statistics, University of California, Berkeley, 2008. [21] S. Roweis and L. Saul, ”Nonlinear dimensionality reduction by locally linear embedding,” Science, vol. 290, no. 5500, pp. 2323– 2326, 2000.