Unsupervised Deep Learning by Neighbourhood Discovery
Jiabo Huang 1 Qi Dong 1 Shaogang Gong 1 Xiatian Zhu 2
Abstract ture representations, which has recently drawn increasing attention (Wu et al., 2018; Caron et al., 2018). Deep convolutional neural networks (CNNs) have demonstrated remarkable success in computer vi- In the literature, representative unsupervised deep learning sion by supervisedly learning strong visual feature methods include clustering (Caron et al., 2018; Xie et al., representations. However, training CNNs relies 2016; Yang et al., 2017) and sample specificity analysis (Wu heavily on the availability of exhaustive training et al., 2018; Bojanowski & Joulin, 2017). The objective of data annotations, limiting significantly their de- clustering is to identify a set of clusters and each repre- ployment and scalability in many application sce- sents an underlying class concept. This strategy has great narios. In this work, we introduce a generic unsu- potential with the best case reaching to the performance pervised deep learning approach to training deep of supervised learning, but is error-prone due to the enor- models without the need for any manual label su- mous combinatorial space and complex class boundaries. pervision. Specifically, we progressively discover In contrast, sample specificity learning avoids the cluster sample anchored/centred neighbourhoods to rea- notion by treating every single sample as an independent son and learn the underlying class decision bound- class. The hypothesis is that the model can reveal the under- aries iteratively and accumulatively. Every single lying class-to-class semantic similarity structure, e.g. the neighbourhood is specially formulated so that all manifold geometry. Whilst collecting such instance labels the member samples can share the same unseen requires no manual annotation cost, the resulting supervi- class labels at high probability for facilitating the sion is ambiguous therefore weak to class discrimination. extraction of class discriminative feature repre- Other contemporary self-supervised learning methods (Do- sentations during training. Experiments on image ersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, classification show the performance advantages 2016; Noroozi et al., 2017; Zhang et al., 2017) share a simi- of the proposed method over the state-of-the-art lar limitation due to the insufficient correlation between the unsupervised learning models on six benchmarks auxiliary supervision and the underlying class target. including both coarse-grained and fine-grained object image categorisation. In this work, we present a generic unsupervised deep learning method called Anchor Neighbourhood Discovery 1. Introduction (AND). The AND model combines the advantages of both clustering and sample specificity learning whilst mitigating Deep neural networks, particularly convolutional neural net- their disadvantages in a principled formulation. Specifi- works (CNNs), have significantly advanced the progress of cally, with a divide-and-conquer principle, the AND dis- arXiv:1904.11567v3 [cs.CV] 30 May 2019 computer vision problems (Goodfellow et al., 2016; LeCun covers class consistent neighbourhoods anchored to indi- et al., 2015). However, such achievements are largely es- vidual training samples (divide) and propagates the local tablished upon supervised learning of network models on a inter-sample class relationships within such neighbourhoods massive collection of exhaustively labelled training imagery (conquer) for more reliably extracting the latent discrimi- data (Krizhevsky et al., 2012a; Dong et al., 2019; 2018). nation information during model training. Neighbourhoods This dramatically restricts their scalability and usability to can be considered as tiny sample anchored clusters with many practical applications with limited labelling budgets. higher compactness and class consistency. They are spe- A natural solution is unsupervised learning of deep fea- cially designed for minimising the clustering errors whilst retaining the exploration of inter-sample class information 1 2 Queen Mary University of London Vision Semantics Limited. that is entirely neglected in sample specificity learning. To Correspondence to: Xiatian Zhu
Nk(x) = {xi | s(xi, x) is top-k in X} ∪ {x}, (1)
where X denotes the feature space. We call such structures as Anchor Neighbourhoods (AN). To enable class discriminative learning, we want all samples in a single neighbourhood AN to share the same class label, i.e. class consistent. As such, we can facilitate the design of learning supervision by assigning the same label to these samples. This requirement, however, is non-trivial to fulfil (a) (b) (c) in unsupervised learning since we have no reasonably good Figure 1. Illustration of three unsupervised learning strategies. (a) sample features, even though a neighbourhood AN can be Clustering analysis aims for discovering the global class deci- much smaller and more local (therefore likely more class sion boundary (Caron et al., 2018; Xie et al., 2016); (b) Sample consistent) than a typical cluster when using small k values. specificity learning discards the concept of clusters by treating Moreover, we begin with the training images but no learned every training sample as an independent class (Wu et al., 2018; features. This even prevents the formation of Nk and gives Bojanowski & Joulin, 2017); (c) Our Anchor Neighbourhood Dis- rise to an extreme case – each individual sample represents covery searches local neighbourhoods with high class consistency. a distinct anchor neighbourhood. To overcome the aforementioned problem, we formulate Neighbourhood Initialisation. Interestingly, such initial an Anchor Neighbourhood Discovery (AND) method. It ANs are in a similar spirit of sample specificity learning takes a divide-and-conquer strategy from the local sample (Wu et al., 2018; Bojanowski & Joulin, 2017) where each anchored neighbourhood perspective. The key idea is that, data instance is assumed to represent a distinct class (Fig whilst it is difficult and error-prone to directly reason the 2(a)). With this conceptual linkage, we exploit the instance global class decision boundaries at the absence of class loss (Wu et al., 2018) to commence the model learning. labels on the training data (Fig1(a)), it would be easier and Specifically, it is a non-parametric variant of the softmax more reliable to estimate local class relationship in small cross-entropy loss written as: neighbourhoods (Fig1(c)). Although such information is incomplete and provides less learning supervision than the nbs > conventional clustering strategy (Caron et al., 2018; Xie X exp(xi xj/τ) Linit = − log(pi,i), pi,j = (2) et al., 2016) that operates at the coarse group level and PN > i=1 k=1 exp(xi xk/τ) mines the clusters of data samples, it favourably mitigates the misleading effect of noisy supervision. Besides, the where n denotes the training mini-batch size, and the tem- proposed AND model differs dramatically from the sample bs perature parameter τ is for controlling the distribution con- specificity learning strategy (Wu et al., 2018; Bojanowski centration degree (Hinton et al., 2014). & Joulin, 2017) that lacks a fundamental ability to mine the inter-sample class relationships primitive to the global Neighbourhood Supervision. In the feature space derived class boundaries (Fig1(b)). Therefore, the proposed method by Eq (2), we build a neighbourhood Nk(x) for each indi- represents a conceptual trade-off between the two existing vidual sample x. Considering the high appearance similarity strategies and a principled integration of them. among the samples of each Nk(x), we assume they share a single class label for model discriminative learning. As shown in our evaluations, the proposed training strategy yields superior models. This indicates the significance of Formally, we formulate an unsupervised neighbourhood Unsupervised Deep Learning by Neighbourhood Discovery
(a) Neighbourhood Similarity (b) Neighbourhood Discovery & Selection Initialisation Distribution Round 1 Round 2 Round R
Low Entropy …
High Entropy
Per-sample Neighbourhoods (c) Neighbourhood Supervision
…
(d) Curriculum Learning
Anchor Neighbourhood …
Training Sample Anchor
Figure 2. Overview of the proposed Anchor Neighbourhood Discovery (AND) method for unsupervised deep learning. (a) The AND model starts with per-sample neighbourhoods for model initialisation. (b) The resulting feature representations are then used to discover the local neighbourhoods anchored to every single training sample, i.e. anchor neighbourhoods. (c) To incorporate the neighbourhood structure information into model learning, we propose a differentiable neighbourhood supervision loss function for enabling end-to-end model optimisation. (d) For enhancing model discriminative learning, we further derive a curriculum learning algorithm for selecting class consistent neighbourhoods in a progressive manner. This is based on a novel similarity distribution entropy measurement. supervision signal as: class consistent. This is because each neighbourhood AN is treated as a different learning concept (e.g. class), although nbs X X some ANs may share the same unknown class label. Such LAN = − log pi,j (3) information is also partial due to that a specific AN may i=1 j∈Nk(xi) represent only a small proportion of the corresponding class, The rationale behind Eq (3) is to encourage label consis- and multiple ANs with the same underlying class can repre- tency for anchor neighbourhoods (Fig2(c)). Specifically, sent different aspects of the same concept collectively (not the whole view due to no AN-to-AN relationships). It is the the probability pi,j (Eq (2)), obtained using a softmax func- set of these distributed anchor neighbourhoods as a whole tion, represents visual similarity between xi and xj in a stochastic manner. This takes the spirit of stochastic near- that brings about the class discrimination capability during est neighbour (Goldberger et al., 2005), as it considers the model training. It is in a divide-and-conquer principle. entire training set. In this scheme, the probability p(x) of Fundamentally, the proposed design differs dramatically correctly classifying a sample xi can be then represented as: from both (1) the clustering strategy that seeks for the com- X plete class boundary information – a highly risky and error- p(xi) = pi,j (4) prone process (Caron et al., 2018; Xie et al., 2016), and (2) j∈C the sample specificity learning that instead totally ignores where C denotes the set of samples in the same class as the class level information therefore less discriminative (Wu xi. However, C is unavailable to unsupervised learning. To et al., 2018; Bojanowski & Joulin, 2017). Moreover, cluster- overcome this problem, we approximate C by the neigh- ing often requires the prior knowledge of the cluster number bourhoods ANs, each of which is likely to be class consistent. therefore limiting their usability and scalability due to the Together with the cross-entropy function, this finally leads lack of it in many applications. On the contrary, this kind to the formulation of the proposed LAN loss (Eq (3)). of information is not needed for forming the proposed ANs, therefore more application generic and scalable. To max- Remarks. The proposed neighbourhood supervision for- imise the class consistency degree in ANs, we simply need mulation L aims at exploring the local class informa- AN to use the smallest neighbourhood size, i.e. k =1. tion, under the assumption that anchor neighbourhoods are Unsupervised Deep Learning by Neighbourhood Discovery
Neighbourhood Selection. As discussed above, the pro- R rounds), we select the top-S (Eq (6)) of ANs according posed method requires the neighbourhoods ANs to be class to their corresponding entropy for model learning by the consistent. This condition, nonetheless, is difficult to meet. proposed neighbourhood supervision loss LAN (Eq (3)). Specifically, the instance loss function Linit (Eq (2)) encour- r ages the feature representation learning towards that each S = ∗ 100% (6) R sample’s specificity degree can be maximised as possible on the training data. Considering a sample xi, other sam- Since the remaining training samples are still not sufficiently ples either share the class label (positive) with xi or not interpreted by the model at the current round, they are pre- (negative). Hence, this formulation may yield a model with served as individuals (i.e. single-sample neighbourhoods) certain discrimination ability, e.g. when a subset of (un- as in sample specificity learning (Eq (2)). known) positive samples are associated with similar visual Objective Loss Function. With the progressive neighbour- specificity. But this entirely depends on the intrinsic data hood discovery as above, we obtain the model objective loss properties without stable guarantee. It means that typically function for the r-th round as: not all neighbourhoods ANs are reliable and class consis- tent. This inevitably leads to the necessity of conducting r X X X L = − log(pi,i) − log pi,j (7) neighbourhood selection for more reliable model learning. r r i∈Binst i∈BAN j∈Nk(xi) To this end, we go beyond by taking advantages of the where Br and Br denote the set of instances and the set curriculum learning idea (Bengio et al., 2009; Dong et al., inst AN of ANs in a mini-batch at the r-th round, respectively. 2017). Instead of taking a one-off neighbourhood selec- tion, we introduce a progressive selection process (Fig2(d)) As each round of training is supposed to improve the model, which distributes evenly the neighbourhood selection across we update the neighbourhoods ANs for all training samples R rounds. This realises an easy-to-hard learning procedure before performing neighbourhood selection per round. To through a curriculum. facilitate this process, we maintain an offline memory to store the feature vectors. We update the memory features of Selecting Curriculum. To enable automated neighbourhood mini-batch samples by exponential moving average (Lucas selection for making a scalable curriculum, it is necessary & Saccucci, 1990) over the training iterations as: for us to derive a selecting criterion. This is achieved by exploiting the intrinsic nature of the probability p (Eq (2)) i,j x˜i = (1 − η) · x˜i + η · xi (8) defined between two samples xi and xj. More specifically, we utilise the entropy measurement of the probability vector where η denotes the update momentum, x and x˜ the up- p = [p , p , ··· , p ] as the class consistency indicator i i i i,1 i,2 i,N to-date and memory feature vector respectively. of the corresponding neighbourhood AN as:
N X 3.2. Model Optimisation H(xi) = − pi,j log(pi,j). (5) j=1 The proposed loss function (Eq (7)) is differentiable there- fore enabling the stochastic gradient descent algorithm for We consider that smaller H(xi) values correspond to more model training. In particular, when xi comes as an individ- consistent neighbourhoods. In particular, when H(xi) is r ual instance, the gradients for L w.r.t. xi and xj (j 6=i) are small, it means xi resides in a low-density area with sparse written as: visual similar neighbours surrounding. In the definition of N sample specificity learning (Eq (2)), the model training tends ∂Lr 1 X ∂Lr 1 = [ (pi,k·xk)+(pi,i−2)·xi], = pi,j ·xi (9) to converge to some local optimum that all samples of a ∂xi τ ∂xj τ k=1 neighbourhood Nk(xi) with small H(xi) share some easy- to-locate visual appearance, and simultaneously the same When xi corresponds to an AN, the gradients are then: underlying class label statistically since positive samples N are more likely to present such appearance commonness ∂Lr 1 X X = [ (pi,k ·xi)− p˜i,k +(pi,i −p˜i,i)·xi] (10) including the context than negative ones. On the contrary, a ∂xi τ k=1 k∈Nk(xi) large H(xi) implies a neighbourhood Nk(xi) residing in a dense area, a case that the model fails to identify the sample r 1 specificity. This is considered hard cases, and requires more ∂L τ [pi,j · xi − p˜i,j · xi], j ∈ Nk(xi) = 1 (11) information for the model to interpret them. ∂xj τ [pi,j · xi], j∈ / Nk(xi) In light of the observations above, we formulate a linear where p˜ = p / P p is the normalised dis- i,j i,j k∈Nk(xi) i,k curriculum according to the class consistency entropy mea- tribution over the neighbours. The whole model training surement. Specifically, for the r-th round (among a total of procedure is summarised in Algorithm1. Unsupervised Deep Learning by Neighbourhood Discovery
Algorithm 1 Neighbourhood discovery. as well as the feature representations extracted from dif- Input: Training data I, rounds R, iterations per round T . ferent network layers respectively. LC was realised by a Output: A deep CNN feature model. fully connected (FC) layer optimised by the cross-entropy Initialisation: Instance specificity learning (Eq (2)). loss function. The non-parametric classifier kNN predicts Unsupervised learning: the class label by weighted voting of top-k neighbours Nk as s = P δ(c, c ) · w where δ(c, c ) is the Dirac for r = 1 to R do c i∈Nk i i i Form neighbourhoods with the current features (Eq (1)); delta function which returns 1 if c = ci, and 0 otherwise. Curriculum selection of neighbourhoods (Eq (6)); The weight wi is computed from the cosine similarity si as for t = 1 to T do wi = exp(si/τ) with τ = 0.07 the temperature parameter. Network forward propagation (batch feed-forward); Without an extra classifier learning post-process involved, Objective loss computation (Eq (7)); kNN reflects directly the discriminative capability of the Network back-propagation (Eq (9),(10),(11)); learned feature representations. Memory feature update (Eq (8)). Performance metric. We adopted the top-1 classification end for accuracy for the model performance measurement. end for Competitors. We compared the proposed AND model with four types of state-of-the-art unsupervised learning meth- ods: (1) Generative model: BiGAN (Donahue et al., 2016); (2) Clustering method: DeepCluster (Caron et al., 2018); (3) Self-supervised learning methods: Context (Doersch (a) CIFAR (b) SVHN (c) ImageNet et al., 2015), Colour (Zhang et al., 2016), Jigsaw (Noroozi & Favaro, 2016), Counting (Noroozi et al., 2017), and Split- Brain (Zhang et al., 2017); (4) Sample specificity learning methods: Instance (Wu et al., 2018) and Noise As Targets (d) CUB200 (e) Dogs (NAT) (Bojanowski & Joulin, 2017); in total 9 methods. Figure 3. Dataset example images. Implementation details. For fair comparisons, we used the 4. Experiments same experimental setting as (Wu et al., 2018; Bojanowski & Joulin, 2017). To train AND models, we set the learning Datasets. We used 6 image classification benchmarks for rate to 0.03 which was further scaled down by 0.1 every evaluating our model (Fig3). CIFAR10(/100) (Krizhevsky 40 epochs after the first 80 epochs. We used the batch size & Hinton, 2009): An image dataset with 50,000/10,000 of 256 for ImageNet and 128 for others. We set the epoch train/test images from 10 (/100) object classes. Each class to 200 per round. We fixed the feature length to 128. We has 6,000 (/600) images with size 32×32. SVHN (Netzer applied the SGD with Nesterov momentum at 0.9. Our et al., 2011): A Street View House Numbers dataset includ- model usually converges with R=4 rounds. We set η =0.5 ing 10 classes of digit images. ImageNet (Russakovsky in Eq (8) for feature update. We set k = 1 (Eq (1)) for et al., 2015): A large 1,000 classes object dataset with 1.2 exploring the most local neighbourhoods. million images for training and 50,000 for test. CUB200- 2011 (Wah et al., 2011): A fine-grained dataset containing 4.1. Comparisons to the State-of-the-Art Methods 5,994/5,794 train/test images of 200 bird species. Stanford Dogs (Khosla et al., 2011): A fine-grained dataset with Small scale evaluation. Table1 compares the object image 12,000/8,580 train/test images of 120 dog breeds. classification results on three benchmarks between AND and four unsupervised learning methods. We tested two Experimental setup. For learning any unsupervised rep- classification models, weighted kNN using FC features and resentation model, we assumed and used only the training linear regression using conv5 features. We have these obser- image data but no class labels. Unless stated otherwise, we vations: (1) The AND method performs best often by large adopted the AlexNet (Krizhevsky et al., 2012b) as the neural margins over all competitors, except linear regression on network architecture for fair comparisons with the state-of- CIFAR10 with DeepCluster outperforms marginally. This the-art methods. To assess the quality of a learned model for suggests the performance superiority of our neighbourhood classification at test time, we utilised the ground-truth class discovery over alternative methods for unsupervised repre- labels of the training images merely for enabling image cate- sentation learning. (2) The margins obtained by kNN tend gorisation. This does not change the feature representations to be larger than those by linear regression. This indicates derived in unsupervised learning. that AND features are favourably more ready for direct use Following Wu et al.(2018), we considered two classifica- without extra classifier training as post-processing. tion models, Linear Classifier (LC), and Weighted kNN, Unsupervised Deep Learning by Neighbourhood Discovery
Dataset CIFAR10 CIFAR100 SVHN strates again the performance superiority of our neighbour- Classifier/Feature Weighted kNN / FC hood discovery idea over the best competitor Instance. Split-Brain∗ 11.7 1.3 19.7 Counting∗ 41.7 15.9 43.4 Dataset CUB200 Dogs DeepCluster 62.3 22.7 84.9 Instance 11.6 27.0 Instance 60.3 32.7 79.8 AND 14.4 32.3 AND 74.8 41.5 90.9 Table 3. Evaluation on fine-grained datasets. Network: ResNet18. Supervised 91.9 69.7 96.5 4.2. Component Analysis and Discussions Classifier/Feature Linear Classifier / conv5 Split-Brain∗ 67.1 39.0 77.3 We conducted detailed component analysis with the Counting∗ 50.9 18.2 63.4 weighted kNN classifier and FC features. DeepCluster 77.9 41.9 92.0 Backbone network. We tested the generalisation of AND Instance 70.1 39.4 89.3 with varying-capacity networks. We further evaluated AND 77.6 47.9 93.7 ResNet18 and ResNet101 (He et al., 2016) on CIFAR10. Supervised 91.8 71.0 96.1 Table4 shows that AND benefits from stronger net architec- Table 1. Evaluation on small scale image datasets. ∗: By a repro- tures. A similar observation was made on ImageNet: 41.2% duced implementation. (ResNet18) vs. 31.3% (AlexNet).
Large scale evaluation. We evaluated the scalability of our Network AlexNet ResNet18 ResNet101 AND model using ImageNet. Table2 compares AND with Accuracy 74.8 86.3 88.4 nine alternative methods. Following the previous studies, we tested all conv layers. The results show that: (1) All unsuper- Table 4. Network generalisation analysis on CIFAR10. vised learning methods clearly surpass the random features, Model initialisation. We tested the impact of initial fea- suggesting their modelling effectiveness consistently. (2) tures for neighbourhood discovery by comparing random AND outperforms all competitors but by smaller margins. and Instance networks. Table5 shows that AND can bene- This is likely due to using over tiny neighbourhoods (sized 2) fit from stronger initialisation whilst being robust to weak for being consistent with small scale datasets. The amount initial representations. of inter-sample relationships is quadratic to the data size, so bigger neighbourhoods may be beneficial for large datasets Initialisation Random Model Instance Model in capturing structural information. (3) Most unsupervised Accuracy 85.7 86.3 learning methods yield the respective best representation at intermediate layers when using linear classifier. The plausi- Table 5. Effect of model initialisation on CIFAR10. ble reason is that their supervision singles are less correlated Neighbourhood size. Neighbourhood size is an important with the ground-truth targets. parameter since it controls label consistency of ANs and finally the model performance. We evaluated its effect using Classifier Linear Classifier kNN ResNet18 on CIFAR10 by varying k from 1 (the default Feature conv1 conv2 conv3 conv4 conv5 FC FC value) to 100. Figure4 shows that the smallest neighbour- Random 11.6 17.1 16.9 16.3 14.1 12.0 3.5 hoods (i.e. k = 1) are the best choice. This implies high Supervised 19.3 36.3 44.2 48.3 50.5 - - variety of imagery data, so smaller ANs are preferred for Context 17.5 23.0 24.5 23.2 20.6 30.4 - unsupervised learning. BiGAN 17.7 24.5 31.0 29.9 28.0 32.2 - Colour 13.1 24.8 31.0 32.6 31.8 35.2 - Curriculum round. We tested the effect of the curriculum Jigsaw 19.2 30.1 34.7 33.9 28.3 38.1 - round (R in Eq (6)) of progressive neighbourhood discovery. NAT - - - - - 36.0 - More rounds consume higher training costs. Figure5 shows Counting 18.0 30.6 34.3 32.5 25.7 - - Split-Brain 17.7 29.3 35.4 35.2 32.8 - 11.8 that using 4 rounds gives a good trade-off between model DeepCluster 13.4 32.3 41.0 39.6 38.2 - 26.8 training efficiency and feature performance. Often, per- Instance 16.8 26.5 31.8 34.1 35.6 - 31.3 round epoch number Nep affects the training efficiency and AND 15.6 27.0 35.9 39.7 37.9 36.7 31.3 performance. We investigated its effect and found that AND Table 2. Evaluation on ImageNet. The results of existing methods achieves 83.3% by Nep=50 vs. 84.8% by Nep=100. are adopted from (Wu et al., 2018; Bojanowski & Joulin, 2017). One-off vs. curriculum neighbourhood discovery. We Fine-grained evaluation. We evaluated AND with more evaluated the benefit of AND’s curriculum. To this end, challenging fine-grained recognition tasks which are sig- we compared with the one-off discovery counterpart where nificantly under-studied in unsupervised learning context. all anchor neighbourhoods are exploited one time. Table6 Consistent with the results discussed above, Table3 demon- shows that the proposed multi-round progressive discovery Unsupervised Deep Learning by Neighbourhood Discovery
45 86.3 40 Consistent 35 30 Inconsistent 83.0 25 20 15 10 Accuracy (%) 79.3 5 0 78.2 1 10 50 100 #Neighbourhoods (x 1000) 1 2 3 4 Neighbourhood Size Training Round Figure 4. Effect of the neighbourhood size on CIFAR10. Figure 6. Neighbourhood quality over rounds on CIFAR10. Image Initial Round 1 Round 2 Round 3 Round 4 86.7
86.3 Black raw r=0 r=1 r=2 r=3 r=4 Stork 85.3 Tiger
Beetle Accuracy (%) Border 84.2 Terrier Black 1 2 4 10 Stork Curriculum Round Figure 5. Effect of the curriculum round on CIFAR10. Tiger Border Beetle Terriervia a curriculum is effective to discover more reliable anchor neighbourhoods for superior unsupervised learning. Coker Coker Spaniel Discovery One-off Curriculum Spaniel Accuracy 84.2 86.3 Gibbon Table 6. One-off vs. curriculum discovery on CIFAR10. Gibon Neighbourhood quality. We examined the class consis- Ladle tency of anchor neighbourhood discovered throughout the curriculum rounds. Figure6 shows that the numbers of both Ladle class consistent and inconsistent anchor neighbourhoods increase along with the training rounds, and more impor- Figure 7. The evolving dynamics of model learning attention tantly the consistent ones raise much more rapidly. This throughout the training rounds on six ImageNet classes. Red explains the performance advantages of the AND model and bounding box indicates a failure case. the benefit of exploring progressive curriculum learning. manner. With the AND model, we avoid the notorious Learning attention dynamics. To further understand how grouping noises whilst still preserving the intrinsic merits of the AND benefits the feature representation learning, we clustering for effective inference of the latent class decision tracked the modelling attention by Grad-Cam (Selvaraju boundaries. Our method is also superior to the existing sam- et al., 2017) to visualise which parts of training images the ple specificity learning strategy, due to the unique capability model is focusing on throughout the curriculum rounds. We of propagating the self-discovered sample-to-sample class have the following observations from Fig7: (1) Often the relationship information in end-to-end model optimisation. model initially looks at class irrelevant image regions. This Extensive experiments on four image classification bench- suggests that sample specificity is a less effective supervi- marks show the modelling and performance superiority of sion signal for guiding model discriminative training. (2) In the proposed AND method over a wide range of state-of-the- cases, the AND model is able to gradually shift the learning art unsupervised deep learning models. We also provided attention towards the class relevant parts therefore yielding in-depth component analysis to give insights on the model a more discriminative model. (3) The AND may fail to cap- advantages of the AND formulation. ture the object attention, e.g. due to cluttered background and poor lighting condition. Acknowledgements 5. Conclusion This work was partly supported by the China Scholarship Council, Vision Semantics Limited, the Royal Society New- In this work, we presented a novel Anchor Neighbourhood ton Advanced Fellowship Programme (NA150459), and Discovery (AND) approach for unsupervised learning of dis- Innovate UK Industrial Challenge Project on Developing criminative deep network models through class consistent and Commercialising Intelligent Video Analytics Solutions neighbourhood discovery and supervision in a progressive for Public Safety (98111-571149). Unsupervised Deep Learning by Neighbourhood Discovery
References Goodfellow, I., Bengio, Y., and Courville, A. Deep learning. MIT press Cambridge, 2016. Aggarwal, C. C. and Reddy, C. K. Data clustering: algo- rithms and applications. CRC press, 2013. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Cur- Conference on computer vision and pattern recognition riculum learning. In Proceedings of the International Con- (CVPR), pp. 770–778, 2016. ference on machine learning (ICML), pp. 41–48, 2009. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowl- Bojanowski, P. and Joulin, A. Unsupervised learning by edge in a neural network. Proceedings of the Conference predicting noise. In Proceedings of the International on Neural Information Processing Systems (NeurIPS), Conference on machine learning (ICML), pp. 1–10, 2017. 2014. Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep Hinton, G. E., Osindero, S., and Teh, Y.-W. A fast learning clustering for unsupervised learning of visual features. In algorithm for deep belief nets. Neural computation, 18 Proceedings of the European Conference on Computer (7):1527–1554, 2006. Vision (ECCV), pp. 1–18, 2018. Kamvar, K., Sepandar, S., Klein, K., Dan, D., Manning, Dizaji, K. G., Herandi, A., Deng, C., Cai, W., and Huang, H. M., and Christopher, C. Spectral learning. In Proceed- Deep clustering via joint convolutional autoencoder em- ings of the International Joint Conference on Artificial bedding and relative entropy minimization. In Proceed- Intelligence (IJCAI). Stanford InfoLab, 2003. ings of the IEEE International Conference on Computer Vision (ICCV), pp. 5747–5756, 2017. Khosla, A., Jayadevaprakash, N., Yao, B., and Fei-Fei, L. Novel dataset for fine-grained image categorization. In Doersch, C., Gupta, A., and Efros, A. A. Unsupervised First Workshop on Fine-Grained Visual Categorization, visual representation learning by context prediction. In IEEE Conference on Computer Vision and Pattern Recog- Proceedings of the IEEE International Conference on nition, Colorado Springs, CO, June 2011. Computer Vision (ICCV), pp. 1422–1430, 2015. Krizhevsky, A. and Hinton, G. Learning multiple layers Donahue, J., Krahenb¨ uhl,¨ P., and Darrell, T. Adversarial of features from tiny images. Technical report, Citeseer, feature learning. In Proceedings of the International 2009. Conference on Learning Representations (ICLR), pp. 1– 18, 2016. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Dong, Q., Gong, S., and Zhu, X. Multi-task curriculum In Proceedings of the Conference on Neural Information transfer deep learning of clothing attributes. In Process- Processing Systems (NeurIPS), 2012a. ings of the Winter Conference on Applications of Com- puter Vision (WACV), pp. 520–529. IEEE, 2017. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Dong, Q., Gong, S., and Zhu, X. Imbalanced deep learn- In Proceedings of the Conference on Neural Information ing by minority class incremental rectification. IEEE Processing Systems (NeurIPS), pp. 1097–1105, 2012b. transactions on pattern analysis and machine intelligence (TPAMI), 2018. Larsson, G., Maire, M., and Shakhnarovich, G. Learning representations for automatic colorization. In Proceed- Dong, Q., Zhu, X., and Gong, S. Single-label multi-class ings of the European Conference on Computer Vision Proceed- image classification by deep logistic regression. (ECCV), pp. 577–593, 2016. ings of the AAAI Conference on Artificial Intelligence (AAAI), 2019. LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436, 2015. Goldberger, J., Hinton, G. E., Roweis, S. T., and Salakhut- dinov, R. R. Neighbourhood components analysis. In Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. Convo- Proceedings of the Conference on Neural Information lutional deep belief networks for scalable unsupervised Processing Systems (NeurIPS), pp. 513–520, 2005. learning of hierarchical representations. In Proceedings of the International Conference on machine learning Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., (ICML), pp. 609–616, 2009. Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Proceedings of the Lucas, J. M. and Saccucci, M. S. Exponentially weighted Conference on Neural Information Processing Systems moving average control schemes: properties and enhance- (NeurIPS), pp. 2672–2680, 2014. ments. Technometrics, 32(1):1–12, 1990. Unsupervised Deep Learning by Neighbourhood Discovery
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Wang, X., He, K., and Gupta, A. Transitive invariance for and Ng, A. Y. Reading digits in natural images with self-supervised visual representation learning. In Proceed- unsupervised feature learning. In NIPS workshop on deep ings of the IEEE International Conference on Computer learning and unsupervised feature learning, number 2, Vision (ICCV), pp. 1329–1338, 2017. pp. 5, 2011. Wu, Z., Xiong, Y., Stella, X. Y., and Lin, D. Unsupervised Ng, A. Sparse autoencoder. pp. 1–19, 2011. feature learning via non-parametric instance discrimina- tion. In Proceedings of the IEEE Conference on computer Noroozi, M. and Favaro, P. Unsupervised learning of visual vision and pattern recognition (CVPR), 2018. representations by solving jigsaw puzzles. In Proceedings of the European Conference on Computer Vision (ECCV), Xie, J., Girshick, R., and Farhadi, A. Unsupervised deep pp. 69–84, 2016. embedding for clustering analysis. In Proceedings of the International Conference on machine learning (ICML) Noroozi, M., Pirsiavash, H., and Favaro, P. Representation , learning by learning to count. In Proceedings of the IEEE pp. 478–487, 2016. International Conference on Computer Vision (ICCV), pp. Yang, B., Fu, X., Sidiropoulos, N. D., and Hong, M. To- 1–9, 2017. wards k-means-friendly spaces: Simultaneous deep learn- Radford, A., Metz, L., and Chintala, S. Unsupervised rep- ing and clustering. In Proceedings of the International resentation learning with deep convolutional generative Conference on machine learning (ICML), pp. 1–14, 2017. adversarial networks. In Proceedings of the International Zhang, R., Isola, P., and Efros, A. A. Colorful image col- Conference on Learning Representations (ICLR), 2016. orization. In Proceedings of the European Conference on Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Computer Vision (ECCV), pp. 649–666, 2016. Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, Zhang, R., Isola, P., and Efros, A. A. Split-brain autoen- M., et al. Imagenet large scale visual recognition chal- coders: Unsupervised learning by cross-channel predic- lenge. International Journal of Computer Vision (IJCV), tion. In Proceedings of the IEEE Conference on computer 115(3):211–252, 2015. vision and pattern recognition (CVPR), pp. 1–11, 2017. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Zhu, X., Loy, C. C., and Gong, S. Constrained clustering: Parikh, D., Batra, D., et al. Grad-cam: Visual explana- Effective constraint propagation with imperfect oracles. tions from deep networks via gradient-based localization. In International Conference on Data Mining (ICDM), pp. In Proceedings of the IEEE International Conference on 1307–1312, 2013. Computer Vision (ICCV), pp. 1–9, 2017. Tang, Y., Salakhutdinov, R., and Hinton, G. Robust boltz- Zhu, X., Loy, C. C., and Gong, S. Constrained clustering mann machines for recognition and denoising. In Pro- with imperfect oracles. IEEE transactions on neural net- ceedings of the IEEE Conference on computer vision and works and learning systems (TNNLS), 27(6):1345–1357, pattern recognition (CVPR), pp. 2264–2271, 2012. 2016. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Man- zagol, P.-A. Stacked denoising autoencoders: Learning useful representations in a deep network with a local de- noising criterion. Journal of Machine Learning Research (JMLR), 11(Dec):3371–3408, 2010. Wagstaff, K., Cardie, C., Rogers, S., Schrodl,¨ S., et al. Con- strained k-means clustering with background knowledge. In Proceedings of the International Conference on ma- chine learning (ICML), volume 1, pp. 577–584, 2001. Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The Caltech-UCSD Birds-200-2011 Dataset. Technical report, 2011. Wang, X. and Gupta, A. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2794–2802, 2015.