Pretraining boosts out-of-domain robustness for pose estimation Alexander Mathis1,2∗† Thomas Biasi2∗ Steffen Schneider1,3 Mert Yuksekg¨ on¨ ul¨ 2,3 Byron Rogers4 Matthias Bethge3 Mackenzie W. Mathis1,2‡ 1EPFL 2Harvard University 3University of Tubingen¨ 4Performance Genetics Abstract Transfer learning (using pretrained ImageNet models), gives a 2X boost on out-of-domain data vs. from scratch training Neural networks are highly effective tools for pose esti- Test (out of domain), mation. However, as in other computer vision tasks, robust- trained from scratch ness to out-of-domain data remains a challenge, especially for small training sets that are common for real-world ap- plications. Here, we probe the generalization ability with three architecture classes (MobileNetV2s, ResNets, and Ef- ficientNets) for pose estimation. We developed a dataset of 30 horses that allowed for both “within-domain” and “out- of-domain” (unseen horse) benchmarking—this is a cru- Test (out of domain), cial test for robustness that current human pose estimation w/ transfer learning benchmarks do not directly address. We show that better MobileNetV2-0.35 EfficientNet-B0 MobileNetV2-1 EfficientNet-B3 ImageNet-performing architectures perform better on both ResNet-50 Train within- and out-of-domain data if they are first pretrained Test (within domain) w/transfer learning on ImageNet. We additionally show that better ImageNet trained from scratch models generalize better across animal species. Further- more, we introduce Horse-C, a new benchmark for common Figure 1. Transfer Learning boosts performance, especially on corruptions for pose estimation, and confirm that pretrain- out-of-domain data. Normalized pose estimation error vs. Ima- ing increases performance in this domain shift context as geNet Top 1% accuracy with different backbones. While training well. Overall, our results demonstrate that transfer learn- from scratch reaches the same task performance as fine-tuning, the ing is beneficial for out-of-domain robustness. networks remain less robust as demonstrated by poor accuracy on out-of-domain horses. Mean ± SEM, 3 shuffles. 1. Introduction eralize to new individuals [39, 30, 34, 42]. Thus, one natu- Pose estimation is an important tool for measuring be- rally asks the following question: Assume you have trained havior, and thus widely used in technology, medicine and an algorithm that performs with high accuracy on a given biology [5, 39, 30, 34]. Due to innovations in both deep (individual) animal for the whole repertoire of movement— learning algorithms [17, 7, 11, 24, 38, 8] and large-scale how well will it generalize to different individuals that have datasets [29, 4, 3] pose estimation on humans has become slightly or dramatically different appearances? Unlike in very powerful. However, typical human pose estimation common human pose estimation benchmarks, here the set- benchmarks, such as MPII pose and COCO [29, 4, 3], con- ting is that datasets have many (annotated) poses per indi- tain many different individuals (> 10k) in different con- vidual (>200) but only a few individuals (≈ 10). texts, but only very few example postures per individual. In To allow the field to tackle this challenge, we developed real world applications of pose estimation, users often want a novel benchmark comprising 30 diverse Thoroughbred to create customized networks that estimate the location horses, for which 22 body parts were labeled by an expert of user-defined bodyparts by only labeling a few hundred in 8, 114 frames (Dataset available at http://horse10. frames on a small subset of individuals, yet want this to gen- deeplabcut.org). Horses have various coat colors and ∗Equal contribution. the “in-the-wild” aspect of the collected data at various †Correspondence: [email protected] Thoroughbred farms added additional complexity. With ‡Correspondence: [email protected] this dataset we could directly test the effect of pretraining 1859 Figure 2. Horse Dataset: Example frames for each Thoroughbred horse in the dataset. The videos vary in horse color, background, lighting conditions, and relative horse size. The sunlight variation between each video added to the complexity of the learning challenge, as well as the handlers often wearing horse-leg-colored clothing. Some horses were in direct sunlight while others had the light behind them, and others were walking into and out of shadows, which was particularly problematic with a dataset dominated by dark colored coats. To illustrate the Horse-10 task we arranged the horses according to one split: the ten leftmost horses were used for train/test within-domain, and the rest are the out-of-domain held out horses. on out-of-domain data. Here we report two key insights: ple different postures per individual (>200) and comprising (1) ImageNet performance predicts generalization for both 30 diverse race horses, for which 22 body parts were labeled within domain and on out-of-domain data for pose estima- by an expert in 8, 114 frames. This pose estimation dataset tion; (2) While we confirm that task-training can catch up allowed us to address within and out of domain general- with fine-tuning pretrained models given sufficiently large ization. Our dataset could be important for further testing training sets [10], we show this is not the case for out-of- and developing recent work for domain adapation in animal domain data (Figure 1). Thus, transfer learning improves pose estimation on a real-world dataset [28, 42, 36]. robustness and generalization. Furthermore, we contrast the domain shift inherent in this dataset with domain shift in- 2.2. Transfer learning duced by common image corruptions [14, 35], and we find pretraining on ImageNet also improves robustness against Transfer learning has become accepted wisdom: fine- those corruptions. tuning pretrained weights of large scale models yields best results [9, 48, 25, 33, 27, 49]. He et al. nudged the field 2. Related Work to rethink this accepted wisdom by demonstrating that for various tasks, directly training on the task-data can match 2.1. Pose and keypoint estimation performance [10]. We confirm this result, but show that on Typical human pose estimation benchmarks, such as held-out individuals (“out-of-domain”) this is not the case. MPII pose and COCO [29, 4, 3] contain many different in- Raghu et al. showed that for target medical tasks (with dividuals (> 10k) in different contexts, but only very few little similarity to ImageNet) transfer learning offers little example postures per individual. Along similar lines, but benefit over lightweight architectures [40]. Kornblith et al. for animals, Cao et al. created a dataset comprising a few showed for many object recognition tasks, that better Im- thousand images for five domestic animals with one pose ageNet performance leads to better performance on these per individual [6]. There are also papers for facial keypoints other benchmarks [23]. We show that this is also true for in horses [41] and sheep [47, 16] and recently a large scale pose-estimation both for within-domain and out-of-domain dataset featuring 21.9K faces from 334 diverse species was data (on different horses, and for different species) as well introduced [20]. Our work adds a dataset comprising multi- as for corruption resilience. 1860 A B 5% training data C 50% training data D Train Normalized distance (nose to eye) Test (within domain) Test (out of domain) X = MobileNetV2 0.35 + = EfficientNet B6 0.3 normalized error 0.5 normalized error Figure 3. Transfer Learning boosts performance, especially on out-of-domain data. A: Illustration of normalized error metric, i.e. measured as a fraction of the distance from nose to eye (which is 30 cm on a horse). B: Normalized Error vs. Network performance as ranked by the Top 1% accuracy on ImageNet (order by increasing ImageNet performance: MobileNetV2-0.35, MobileNetV2-0.5, MobileNetV2-0.75, MobileNetV2-1.0, ResNet-50, ResNet-101, EfficientNets B0 to B6). The faint lines indicate data for the three splits. Test data is in red, train is blue, grey is out-of-domain data C: Same as B but with 50% training fraction. D: Example frames with human annotated body parts vs. predicted body parts for MobileNetV2-0.35 and EfficientNet-B6 architectures with ImageNet pretraining on out-of-domain horses. What is the limit of transfer learning? Would ever larger 3. Data and Methods data sets give better generalization? Interestingly, it appears to strongly depend on what task the network was pretrained 3.1. Datasets and evaluation metrics on. Recent work by Mahajan et al. showed that pretraining We developed a novel horse data set comprising 8, 114 for large-scale hashtag predictions on Instagram data (3.5 frames across 30 different horses captured for 4 − 10 sec- billion images) improves classification, while at the same onds with a GoPro camera (Resolution: 1920×1080, Frame time possibly harming localization performance for tasks Rate: 60 FPS), which we call Horse-30 (Figure 2). We like object detection, instance segmentation, and keypoint downsampled the frames by a factor of 15% to speed-up detection [32]. This highlights the importance of the task, the benchmarking process (288 × 162 pixels; one video rather than the sheer size as a crucial factor. Further cor- was downsampled to 30%). We annotated 22 previously roborating this insight, Li et al. showed that pretraining on established anatomical landmarks for equines [31, 2]. The large-scale object detection task can improve performance following 22 body parts were labeled in 8, 114 frames: for tasks that require fine, spatial information like segmen- Nose, Eye, Nearknee, Nearfrontfetlock, Nearfrontfoot, Of- tation [27]. Thus, one interesting future direction to boost fknee, Offfrontfetlock, Offfrontfoot, Shoulder, Midshoul- robustness could be to utilize networks pretrained on Open- der, Elbow, Girth, Wither, Nearhindhock, Nearhindfet- Images, which contains bounding boxes for 15 million in- lock, Nearhindfoot, Hip, Stifle, Offhindhock, Offhindfet- stances and close to 2 million images [26].