IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. X, XX 2018 1 Image Segmentation Based on Multiscale Fast Spectral Clustering Chongyang Zhang, Guofeng Zhu, Minxin Chen, Hong Chen, Chenjian Wu Abstract—In recent years, spectral clustering has become one In recent years, researchers have proposed various ap- of the most popular clustering algorithms for image segmenta- proaches to large-scale image segmentation. The approaches tion. However, it has restricted applicability to large-scale images are based on three following strategies: constructing a sparse due to its high computational complexity. In this paper, we first propose a novel algorithm called Fast Spectral Clustering similarity matrix, using Nystrom¨ approximation and using based on quad-tree decomposition. The algorithm focuses on representative points. The following approaches are based the spectral clustering at superpixel level and its computational on constructing a sparse similarity matrix. In 2000, Shi and 3 complexity is O(n log n) + O(m) + O(m 2 ); its memory cost Malik [18] constructed the similarity matrix of the image by is O(m), where n and m are the numbers of pixels and the using the k-nearest neighbor sparse strategy to reduce the superpixels of a image. Then we propose Multiscale Fast Spectral complexity of constructing the similarity matrix to O(n) and Clustering by improving Fast Spectral Clustering, which is based to reduce the complexity of eigen-decomposing the Laplacian on the hierarchical structure of the quad-tree. The computational 3 complexity of Multiscale Fast Spectral Clustering is O(n log n) matrix to O(n 2 ) by using the Lanczos algorithm. However, and its memory cost is O(m). Extensive experiments on real its computational complexity and memory cost are still high large-scale images demonstrate that Multiscale Fast Spectral when their method is applied to large-scale images. To further Clustering outperforms Normalized cut in terms of lower compu- reduce the computation time and memory cost, in 2005, tational complexity and memory cost, with comparable clustering accuracy. T. Cour et al. [19] used multiscale graph decomposition to construct the similarity matrix. The computational complexity Index Terms—Image segmentation, multiscale, quad-tree de- of this algorithm is linear in the number of pixels. Some composition, spectral clustering, superpixel. researchers proposed several approaches based on Nystrom¨ approximation. In 2004, C. Fowlkes et al. [20] presented the I. INTRODUCTION method based on Nystrom¨ approximation, in which only a small number of random samples were used to extrapolate the LUSTERING is an important method of data processing complete grouping solution. The complexity of this method is with a wide range of application such as topic modeling 3 C O(m )+O(m1n), where m1 represents the number of sample 1 [1], image processing [2], [3], medical diagnosis [4] and pixels in the image. However, deterministic guarantee on the community detection [5]. and applied to imagesegmentation. clustering performance cannot be provided by random sam- A variety of clustering algorithms have been developed so pling [10]. In 2017, Zhan Qiang and Yu Mao [10] improved far, including prototype-based algorithm [6], density-based the algorithm of spectral clustering based on incremental algorithm [7], graph theory-based algorithm [8], etc. The k- Nystrom¨ by the Nystrom¨ sampling method. Computational means algorithm [9], a prototype-based algorithm, has the ad- 2 2 complexity was reduced to O(n )+O(Mm1+nm1)+O(knt), vantage of low computational complexity. However, it doesn’t where k represents the number of clusters, t represents the work well on non-convex data sets. Density-Based Spatial number of the iterations of k-means and M is a constant. The Clustering of Applications with Noise (DBSCAN) is a typical following approaches are based on representative points. In density-based algorithm, but it costs a large amount of mem- 2009, Yan et al. [21] proposed the k-means-based approximate arXiv:1812.04816v1 [eess.IV] 12 Dec 2018 ory. Spectral clustering algorithms based on the graph theory spectral clustering method. First, The image is partitioned into are appropriate for processing non-convex data sets [10], [11] some superpixels by k-means. Then, the traditional spectral though, it is difficult to be applied to large-scale images due clustering is applied to the superpixels. The computation to its high computational complexity [1], [12]–[16], which is time of the method is O(k3) + O(knt). In 2015, Cai et al. primarily caused by two procedures: 1) construction of the [22] proposed a scalable spectral clustering method called similarity matrix, and 2) eigen-decomposition of the Laplacian Landmark-based Spectral Clustering (LSC). LSC generates p matrix [17]. The computational complexity of procedure 1) is representative data points as the landmarks and uses the linear 2 3 O(n ) and that of 2) O(n ), an unbearable burden for the combinations of those landmarks to represent the remaining segmentation of large-scale images. data points. Its computational complexity scales linearly with the size of problem. This work was supported by the National Natural Science Foundation of China under Grant 61801321. In this paper, we first propose a novel spectral clustering Chongyang Zhang and Chenjian Wu are affiliated with the School of algorithm for large-scale image segmentation based on super- Electronic and Information Engineering, Soochow University, Suzhou, China. pixels called Fast Spectral Clustering (FSC). Then we enhance Guofeng Zhu, Minxin Chen and Hong Chen are affiliated with the School of Mathematical Sciences, Soochow Univerity, Suzhou, China. the method and present Multiscale Fast Spectral Clustering The corresponding author: Chenjian Wu, E-mail: [email protected] (MFSC), which is based on the hierarchical structure of the IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. X, XX 2018 2 quad-tree. A brief introduction to MFSC: The superpixels To solve the above problem, the matrix X = (xij)n×k is of image I are obtained by quad-tree decomposition during defined as follows: which the hierarchical structure of the quad-tree is reserved. ( 1 We propose a “bottom up” approach: along the hierarchical p vi 2 Cj, vol(C ) structure of the quad-tree, we merge child nodes at the fine xij = j (3) 0 otherwise: level into their parent node at the coarse level by treating ¯ the clusters, the segmentation result of child nodes, as the T cut(Cj ,Cj ) T It is easy to verify x (D−W )xj = and X DX = superpixels of the parent node. The computational complexity j vol(Cj ) of the algorithm is O(n log n) and its memory cost is O(m). E, where E is an identity matrix and the degree matrix D is d = Pn w The reminder of the paper is organized as follows. In defined as the diagonal matrix whose entry is i j ij, Section II, we introduce the preliminaries to the formulation degree of vi. of our algorithms from the aspects of Ncut and quad-tree Next, the unnormalized graph Laplacian L is defined as decomposition. In Section III, we describe our two algorithms follows: FSC and MFSC and their respective complexity in detail. L = D − W: (4) Experimental results are shown in Section IV. Finally, we conclude our work in Section V. With matrices X and L, the minimization problem in Eq. (2) can be rewritten as the following problem: II. PRELIMINARIES min T r(XT LX) A. Normalized Spectral Clustering C (5) s:t: XT DX = E: This section gives a brief introduction to K-way Normalized cut (Ncut) proposed by Shi et al. [18]. Suppose image I Then, relaxing the discreteness condition and substituting Y = 1 contains pixels v1, ::: ,vn, and the similarity matrix of image D 2 X, the following relaxed problem is obtained : I is the matrix W = (wij)n×n, in which wij denotes the T similarity between pixel vi and pixel vj [23]. According to T. min T r(Y LN Y ) Y 2Rn×k Cour et al. [19], wij is defined as follows: (6) s:t: Y T Y = E, p 2 2 wI (i,j) × wC (i,j) + αwC (i,j) kXi − Xj k ≤ r , wij = 0 otherwise: where (1) where 1 1 − 2 − 2 2 2 LN = D (D − W )D (7) − kXi−Xj k /σx−kZi−Zj k /σI wI (i,j) = e , 2 min −kEdge(x) k /σC is a normalized graph Laplacian. Eq. (6) is the standard form of x2line(i,j) wC (i,j) = e , a trace minimization problem. The Rayleigh-Ritz theorem [26] where Xi and Zi denote the location and intensity of pixel tells us that its solution is the matrix whose columns are the vi; r denotes graph connection radius; σx and σI are scaling first k eigenvectors of matrix LN (By “the first k eigenvectors” parameters; Edge(x) is the edge strength at location x; we refer to the eigenvectors corresponding to the k smallest line(i,j) is the straight line connecting pixels vi and vj [19]. eigenvalues). Also, it is obvious that solution X consists of If the straight line connecting the two pixels does not cross the first k generalized eigenvectors of Lu = λDu [24]. the edge of the image, the value of wC will be large, reflecting The algorithm of normalized spectral clustering by Shi and that the affinity of the two pixels is high. With the similarity Malik [18] is presented in Algorithm 1. Its computational 3 matrix W , K-way Ncut clusters the image into k clusters complexity is O(n 2 ); its memory cost is O(n). C = fC1, C2,:::,Ckg by solving the following minimization problem [18], [24], [25]: Algorithm 1 Normalized spectral clustering according to Shi and Malik [18], [27] min Ncut(C), (2) C Input: The similarity matrix W and the number of desired where clusters k.