Pattern Recognition 43 (2010) 4069–4076 Contents lists available at ScienceDirect Pattern Recognition journal homepage: www.elsevier.com/locate/pr Enabling scalable spectral clustering for image segmentation Frederick Tung Ã, Alexander Wong, David A. Clausi Vision and Image Processing Lab, Systems Design Engineering, University of Waterloo, 200 University Ave. West, Waterloo, Ontario, Canada N2L 3G1 article info abstract Article history: Spectral clustering has become an increasingly adopted tool and an active area of research in the Received 1 February 2010 machine learning community over the last decade. A common challenge with image segmentation Received in revised form methods based on spectral clustering is scalability, since the computation can become intractable for 25 May 2010 large images. Down-sizing the image, however, will cause a loss of finer details and can lead to less Accepted 20 June 2010 accurate segmentation results. A combination of blockwise processing and stochastic ensemble consensus are used to address this challenge. Experimental results indicate that this approach can Keywords: preserve details with higher accuracy than comparable spectral clustering image segmentation Spectral clustering methods and without significant computational demands. Image segmentation & 2010 Elsevier Ltd. All rights reserved. Stochastic ensemble consensus 1. Introduction 2. Spectral clustering In the last decade, spectral clustering has become a very active Like other clustering algorithms, spectral clustering attempts area of research in the machine learning community, with many to partition data points into groups such that the members extensions and applications of the algorithm being developed. of a group are similar to each other and dissimilar to data One of the more popular applications of spectral clustering points outside of the group. Spectral clustering has a simple algorithms is image segmentation. Image segmentation plays a formulation and can be solved by standard linear algebra fundamental role in computer vision as a requisite step in techniques, however, it typically produces better results than such tasks as object detection, classification, and tracking [1]. traditional clustering algorithms such as k-means and mixture Application areas of image segmentation include content-based models [4–7]. image retrieval, automated industrial inspection, medical image Spectral clustering requires the construction of a weighted processing, and remote sensing [2,3]. graph that encodes the similarity (or affinity) between data One of the challenges of image segmentation algorithms based points. Nodes in the graph correspond to data points; the weight on spectral clustering is scalability. Spectral clustering involves of the edge between two nodes is a function of the similarity the eigendecomposition of a pairwise similarity matrix, which is between the corresponding two data points. Given data points intractable for sufficiently large images. Down-sizing the image, x1,y,xn, this weighted graph can be represented by a weighted however, will cause a loss of finer details and can lead to adjacency matrix W, where wij is a measure of the similarity inaccurate segmentation results. The proposed method solves this between xi and xj. In the case of image segmentation, n is the total problem by successfully applying spectral clustering to large number of pixels in the image. images using a combination of blockwise processing and The degree di of node i is the sum of all edge weights incident stochastic ensemble consensus. on xi: Section 2 briefly outlines the basic spectral clustering Xn algorithm. Section 3 discusses related work in spectral clustering di ¼ wij ð1Þ image segmentation. Section 4 describes the proposed method for j ¼ 1 scalable and detail-preserving spectral clustering image segmen- tation. Experimental results are presented in Section 5, and The degree matrix D is defined as the n  n diagonal matrix with conclusions and directions for future work are presented in d1,y,dn on the diagonal. Finally, the graph Laplacian matrix can Section 6. be defined as [5,8] L ¼ DÀW ð2Þ Ã Corresponding author. Tel.: +1 519 888 4567x35342; fax: +1 519 746 4791. E-mail addresses: [email protected] (F. Tung), Note that this is just one possible definition of the graph Laplacian [email protected] (A. Wong), [email protected] (D.A. Clausi). and variants do exist (e.g., as found in [4]). 0031-3203/$ - see front matter & 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.patcog.2010.06.015 4070 F. Tung et al. / Pattern Recognition 43 (2010) 4069–4076 The basic spectral clustering algorithm can now be formulated maximization (EM) algorithm. Instead of the final k-means step, as follows [5,8]: the authors propose fitting a Gaussian mixture model in which Given: Data points x1,y,xn, number of clusters k the number of mixture components determined by the Bayesian Information Criterion [13]. Applications in both image segmenta- 1. Calculate a weighted similarity graph over the data points, tion and behaviour clustering in video are demonstrated. 2. Calculate the Laplacian matrix L ¼ DÀW, where D is the Fowlkes et al. [14] explicitly addressed the scalability issue degree matrix and W is the weighted adjacency matrix by applying the Nystrom¨ approximation to the normalized induced by the similarity graph, as described above, cut framework [8]. Their method is able to efficiently find an 3. Calculate the first k eigenvectors (the eigenvectors correspond- approximate segmentation using the pairwise similarities ing to the k smallest eigenvalues) v1,y,vk of the generalized between a small random subset of the image pixels and the eigenproblem Lv ¼ lDv, where 1rkrn, entire image. Segmentation results are visually comparable to 4. Construct a matrix V with eigenvectors v1,y,vk as columns, those obtained by normalized cut. k 5. Create a new set of points y1,y,yn, such that yi AR is the ith row of V, 6. Cluster the new set of points y1,y,yn using k-means [9] to 4. Proposed method obtain clusters C1,y,Ck. The proposed method addresses a common challenge faced by Return: Set of clusters A1,y,Ak, in which xj is assigned to Ai if yj is image segmentation algorithms based on spectral clustering: assigned to Ci. scalability. To address this challenge, the proposed method In the case of image segmentation, L and D are n  n matrices. combines a blockwise segmentation strategy with stochastic Solving the generalized eigenproblem can easily become intract- ensemble consensus. The overarching idea is to perform an able, for even small images. A larger 1000  1000 image would over-segmentation of the image at the pixel level using spectral involve 106  106 matrices. The operation is commonly made clustering, and then merge the segments using a combination of tractable by considering only each pixel’s local spatial neighbour- stochastic ensemble consensus and a second round of spectral hood in the construction of the weighted similarity graph; that is, clustering at the segment level. The purpose of using stochastic a pixel has zero similarity to pixels outside its local neighbour- ensemble consensus is to integrate both global and local image hood. This produces sparse W, D, and L matrices, that allows an characteristics in determining the pixel classifications. The efficient algorithm for approximating only the first k eigenvalues stochastic ensemble consensus step also removes blockwise [8]. However, for sufficiently large images and neighbourhood processing artifacts. The rest of this section elaborates on the sizes, this computation can still be impractical in terms of specifics of the algorithm. memory requirements. The interested reader can find a more complete treatment of 4.1. Blockwise segmentation the theory behind spectral clustering in [10]; a good introduction to spectral clustering can also be found in [5]. In the first step of the algorithm, the image is partitioned into non-overlapping blocks of fixed size 32  32. An over-segmenta- 3. Related work tion of the image is performed by segmenting each 32  32 block in sequence using spectral clustering. In essence, each 32  32 block is treated as a separate image. For each block, a 322  322 ¼ Possibly the first image segmentation algorithm based on 1024  1024 similarity matrix is calculated over the pixels. The spectral clustering was developed by Shi and Malik [8]. Shi and similarity between a pixel x and another pixel x in the Malik formulated the problem from a graph theoretic perspective i j neighbourhood of x is determined by an exponentially decaying and introduced the normalized cut to segment the image. Malik i function of the squared difference in their intensity values: et al. applied the normalized cut formulation to segment ( grayscale images based on a combination of contour and texture 2 expðaðIðxiÞIðxjÞÞ Þ xj is in the neighbourhood of xi sij ¼ ð3Þ cues [11]. 0 otherwise Zelnik-Manor and Perona [7] proposed a method for auto- matically determining an appropriate number of clusters where I(x) denotes the intensity of pixel x and a is a scaling (segments) and adaptively selecting an appropriate neighbour- constant that controls a penalty based on intensity difference. The hood size or scale. The number of clusters is determined by constant a is set to 30 in all experiments. Empirically, values in minimizing the cost, for a range of possible numbers of clusters, the range of 25–30 tend to be effective and produce similar associated with rotating the columns of V in alignment with a results. As described in Section 2, pixels outside of the local canonical coordinate system, such that every row of the rotated neighbourhood are assigned similarity values of zero to allow for V contains at most one non-zero entry. A local neighbourhood a sparse representation of the similarity matrix. A neighbourhood scale is determined for each pixel based on the distance to its kth size of 9  9 is used in all experiments.