Deep Metric Learning with Tuplet Margin Loss
Total Page:16
File Type:pdf, Size:1020Kb
Deep Metric Learning with Tuplet Margin Loss Baosheng Yu and Dacheng Tao UBTECH Sydney AI Centre, School of Computer Science, Faculty of Engineering, The University of Sydney, Darlington, NSW 2008, Australia {baosheng.yu, dacheng.tao}@sydney.edu.au Abstract ¯ =0 ¯>0 x Deep metric learning, in which the loss function plays a p ¢ x ) ni key role, has proven to be extremely useful in visual recog- ¯ ¡ xa xa ap nition tasks. However, existing deep metric learning loss µ functions such as contrastive loss and triplet loss usually cos( x ¡ n i x i an p rely on delicately selected samples (pairs or triplets) for fast µ cos ¡ convergence. In this paper, we propose a new deep metric s learning loss function, tuplet margin loss, using randomly e 1 selected samples from each mini-batch. Specifically, the proposed tuplet margin loss implicitly up-weights hard sam- Easy 0 Hard ples and down-weights easy samples, while a slack margin cosµan cos(µap ¯) in angular space is introduced to mitigate the problem of i ¡ ¡ overfitting on the hardest sample. Furthermore, we address the problem of intra-pair variation by disentangling class- specific information to improve the generalizability of tu- Figure 1: An illustration of tuplet margin loss function. plet margin loss. Experimental results on three widely used Given a tuplet (xa,xp,xn1 ,...,xnk−1 ), tuplet margin loss deep metric learning datasets, CARS196, CUB200-2011, exponentially up-weights hard triplets and down-weights and Stanford Online Products, demonstrate significant im- easy triplets within the tuplet. Specifically, the loss of each provements over existing deep metric learning methods. triplet (xa,xp,xni ) is defined by the scale factor s> 1 and the violate margin cos θani −cos θap. A slack margin β > 0 is used to mitigate the problem of overfitting on the hard- 1. Introduction est triplets by paying more attention to “moderately hard triplets” (the shaded area). See more details in Section 3.3. Deep metric learning focuses on learning a deep feature embedding consistent with semantic similarity, i.e., a small intra-class variation and a large inter-class variation [38, backs. Specifically, classification-based loss functions usu- 35]. It has been proven that deep metric learning methods ally use a classification layer or a reference point for each are extremely valuable in visual recognition tasks such as class [36], in which both the computation and the require- one-shot learning [5, 29], image retrieval [9, 18], person re- ments on device memory increase linearly with the num- identification [39, 12], and face recognition [27, 23]. With ber of classes [8]. Recently, several methods such as dy- the growing scale of training data, i.e., both the number namic class selection [43] and distributed parallel accelera- of samples and classes, deep metric learning loss function tion [4] have been developed to relieve the computation and has attracted more and more attention in large-scale visual memory bottlenecks in classification-based loss functions, recognition tasks [23, 17]. while the discussion of approximation algorithms for mas- Deep metric learning loss function can be divided into sive classification is beyond the scope of this paper. Re- two main groups: (1) classification-based loss functions, gardless of the heavy classification layer or massive ref- e.g., large-margin softmax loss [14] and center loss [36]; erence points, distance-based loss functions directly opti- and (2) distance-based loss functions, e.g., contrastive mize the margin between intra- and inter-class distances, loss [2, 27] and triplet loss [24, 23]. However, existing and are independent with the number of classes [23]. How- loss functions usually suffer from several inherent draw- ever, existing distance-based loss functions, e.g., triplet loss, 16490 ing only a set of randomly sampled tuplets. Inspired by the 푥 푥 푥# 푥$ 푥 푥% $& & & # # # focal loss for object detection [13], we exponentially up- weight hard triplets and down-weight easy triplets within 푥 푥 푥 푥$ 푥 푥% $$ $ #$ $ $ $ each tuplet (see an example in Figure 1). However, the ex- ponential weighting scheme usually tends to form a rela- tively large margin between the intra- and inter-class dis- 휎 휏 ! tances by overfitting the hardest triplet in each tuplet. To 휎 " solve this problem, we introduce a slack margin in angular (a) (b) space to pay more attention to “moderately hard triplets” rather than “the hardest triplets” [23]. An intuitive example of the slack margin for changing the weighting scheme is Figure 2: An illustration of intra-pair variation. The height shown in Figure 1. of each bar indicates pairwise distance, i.e., color-fill bar Distance-based loss functions, including the proposed for positive pair and pattern-fill bar for negative pair. In tuplet margin loss, focus on optimizing the margin between both (a) and (b), there is a clear margin between positive intra- and inter-class distances by penalizing the margin and negative pairs, i.e., we have ∀i = 1, 2, d(xai ,xpi ) < between positive and negative pairs with the same anchor . However, in (a), the distance metric on each d(xai ,xni ) point, while leaving a risk of learning a class-dependent dis- specific type of pairs (positive or negative pairs) varies tance metric from the training set. A class-dependent dis- among different classes, i.e., the distribution of pairwise tance metric indicates that the distance distribution of posi- distance is class-dependent on training set. Comparing with tive pairs (or negative pairs) varies among different classes, the class-independent distribution shown in (b), the intra- which has not been well-addressed in previous work [1]. pair variation increases the risk of failing to find a proper We refer to the variation within each type of pairs (pos- threshold τ to separate all positive and negative pairs and itive or negative pairs) as the intra-pair variation, and ar- degrades the performance for visual recognition such as ver- gue that the intra-pair variation degrades the generalizabil- ification task. A class-independent distance metric can be ity of deep metric learning model. An intuitive failure case learned by minimizing the intra-pair variances, i.e., σap for induced by the intra-pair variation is shown in Figure 2. positive pairs and σan for negative pairs. See more details To solve this problem, we disentangle the intra-pair vari- in Section 3.4. ation from intra/inter-class variation and try to learn a class- independent distance metric by minimizing the intra-pair variances in both positive and negative pairs. usually suffer from the problem of slow convergence, and In this paper, our main contribution is a new tuplet-based rely heavily on mining informative samples for fast conver- deep metric learning loss function: (1) we propose a tuplet gence [23, 8], raising a number of severe sampling prob- margin loss by using a set of randomly selected samples, lems: (1) the number of possible triplets grows cubically which is computationally more efficient than explicitly min- with the number of training samples [8]; (2) mining in- ing informative samples; (2) we introduce a slack margin to formative triplets tends to be difficult, e.g., both randomly address the problem of overfitting on the hardest sample; selected triplets and the hardest triplets lead to bad local and (3) we address the problem of intra-pair variation to minima [23, 6]; and (3) the training stability benefits from further improve the generalizability of deep metric learn- large mini-batches, in which cross-device synchronization ing model. Specifically, with the proposed tuplet margin is a non-trivial engineering task [23, 17]. loss, we achieve the state-of-the-art results on three widely Recently, significant improvements on distance-based used deep metric learning datasets, i.e., CARS196 [11], loss function have been achieved by using the notation of CUB200-2011 [30], and Stanford Online Products [18]. tuplet, which generalizes a triplet with multiple negative examples to form a better approximation of inter-class dis- 2. Related Work tance [18, 25, 17]. Although delivering impressive perfor- mance improvements, tuplet-based loss functions further Classification-based Loss Function. Feature embeddings exacerbate the sampling problems, because the computa- learned by the classification loss generalize well to a vari- tional complexity exponentially increases with the number ety of visual recognition tasks [21, 40]. Inspired by this, of negative examples. Therefore, here we develop a new center loss [36] and large-margin softmax loss [14] have tuplet-based loss function, tuplet margin loss, using a set of been proposed to further improve the discriminability of randomly sampled tuplets. Unlike previous distance-based classification-based loss function. Specifically, center loss loss functions [18, 25], in which informative samples are minimizes the distance between each example and its class explicitly selected by sampling heuristics, we address infor- center, forming a class-dependent constraint. Large-margin mative samples from the view of loss function, while us- softmax loss has since been significantly improved by both 6491 푛 512푑 푥- 푥. 푥/0 푥/1 * * 푥/230 1 2 + ⋯ + * + ⋯ + 푘 … ⋯ * 푘푛(푛 − 1) + ⋯ + Batch CNNs Embedding Random Tuplet Selection Tuplet Margin Loss Figure 3: An illustration of the tuplet-based deep metric learning framework. We first randomly sample a mini-batch of training data, which contains kn training samples from k different classes, i.e., n samples per class. The deep neural network is used to learn a fixed-dimensional feature embedding, e.g, 512d. We then construct a set of tuplets using all kn(n − 1) positive pairs within the mini-batch and negative examples are randomly sampled from each of the other k − 1 classes. Finally, the loss function is evaluated on the tuplets constructed from each mini-batch.