Supplementary This supplementary document provides additional qual- itative and quantitative results that provide further insights into the results discussed in the main paper. A more de- tailed description of the datasets is given in I and the pose § clusters discussed in 3.1 of the main paper are visualized § in II. Additional experimental results and visualizations § that show the effectiveness of different components of our framework are given in III and IV, respectively. § § I. Datasets I.1. IMDB We created the IMDB database to train the actor clas- sifier in the movie scenario. The scenario is different from most of the person recognition in that the test set contains a single movie with lesser variation in appearance between multiple instances of an actor in terms of age, style of cloth- ing, etc. We assume that there are no labeled images within the movie and hence training data is not a part of the movie. Figure 12: IMDB: Each row shows few images of an actor To create a training set, the images are collected from the from the dataset. We used IMDB dataset to train classifiers IMDB profile2 of actors appearing in the movie, which are for actor recognition in the Hannah movie. then manually cropped and annotated. Few images from IMDB database are shown in Figure 12. We relied on text tags associated with photos for annotation whenever the photos contain multiple confusing identities. Apart from il- lumination, resolution, and pose variations, there is a large age variations among IMDB instances. In addition, there is a large domain contrast between IMDB and Hannah test set in terms of lighting, camera and imaging conditions. This creates a more challenging setting to match identites between IMDB and Hannah instances. I.2. Soccer Soccer is another scenario where there are a significant number of frames in which the face is not visible and the subjects are often occluded by other players. We show more examples from our soccer dataset in Figure 14. In many in- stances, head is largely occluded, and in back-view unlike PIPA and Hannah instances, which contain visible head and torso regions. Also, soccer instances exhibit large body de- formations, are of low resolution with significant blur. The soccer dataset therefore offers different kinds of challenges for recognition that are not seen in PIPA and Hannah.
II. Pose clusters We obtain a set of prominent views to facilitate pose- specific representations as discussed in 3.1. To achieve Figure 13: Pose clusters: Each row from top to bottom § this, we annotated 14 body keypoints for 29,223 PIPA shows people from PIPA with particular body orientation train instances which are then used for clustering. More clustered using orientation and keypoint visibility features.
2http://www.imdb.com/title/tt0091167/ fullcredits Figure 14: Images from soccer dataset. It offers a challenging person recognition scenario due to low resolution, high occlusion, deformation and motion blur exhibited by soccer instances. examples of our pose clusters are shown in Figure 13. III. Quantitative Results and Analysis Each row from top to bottom contain images from right, semi-right, frontal, semi-left, left, back We provide more insightful results that help to under- and partial body views. The orientation and keypoint stand merits and challenges of different recognition settings visibility features produced tight clusters containing images that are considered. with particular body orientation. The last cluster captures Recognition per subject: Figure 15 shows the number the instances with partial upper body such as head or shoul- of images for each actor in IMDB and Hannah test sets der, etc, in the images that are commonly seen in social me- along with their individual recognition performances. We dia photos and movies. While we considered seven promi- observe that, for those subjects with sufficiently large num- nent views in this work, we note that generating a large ber of training instances (Michael Caine, Barbara Harshey, number of views can be helpful, provided there are enough Woody Allen, Julia Louis-Dreyfus, and Mia Farrow), the training samples in each cluster to train the convnets. performance is high as expected. For subjects with less than 20 training instances, the performance is very low. etn.Fratr ihls hnei perneoe time over appearance movie in the ( change in less useful with less actors are For color setting. hair and glasses, attributes human age, and like scene as such clues photo-albums, like with than accu- an of reaches approach racy Our respectively. 18, Figure and on approaches various 5 of performance recognition the pare next. discussed are which cues, clothing to due near a ( observe keepers also We ( stances. subjects see for We performance high Huguain 16. of Figure trend similar in a performances individual their with examples. training enough be- age Jenkins ( in Richard instances difference test large and train a tween is there whenever However, set. test the on (bottom) actor the each show of We performance set. recognition IMDB test (top) movie in Hannah actor (middle) each for and images of Number 15: Figure ihe Caine Michael otocrigmveadsce ujcsi iue17 Figure in subjects soccer and movie occurring most eonto efrac ftpsubjects: top of performance Recognition along players soccer of statistics the show we Similarly, #images in Hannah movie naeil 17
Recognition accuracy 12000 18000 24000 30000 #images in IMDB 6000 100 100 200 300 400 20 40 60 80 61 oesi oprbet edaduprbd.Un- body. upper and head to comparable is models aulNeuer Manuel and
. Carrie Fisher 17
oeta h vrl efrac of performance overall the that Note . Joanna Gleason ntpatr,wihi infiatybetter significantly is which actors, top on %
org Palacio Rodrigo Sam Waterston
and Julia Louis-Dreyfus ,tepromnei ordsiehaving despite poor is performance the ), Verna O. Hobson Woody Allen od Allen Woody Maureen OSullivan Tony Roberts and Lewis Black Paul Bates
egoRomero Sergio Fred Melamed Richard Jenkins ihsfcettann in- training sufficient with ) areFse,Dan West, Dianne Fisher, Carrie Daniel Stern
e o he n v in five and three row See : J.T. Walsh 100 Michael Caine John Turturo
cuayfrgoal for accuracy % Lloyd Nolan Ken Costigan Max Von Sydow n h referee the and ) Julie Kavner Barry Gibb Mia Farrow Diane Wiest ecom- We
Gonzalo Christian Clemenson naeil Bobby Short Barbara Hershey fps wr oesand models to performance aware overall flexibility pose the of more Finally, provide regions. selective robust it on more as focus is better model much trained performs jointly and using concatena- features the hand, of other tion the On alone. than feature worse body is upper training separate and through head obtained of body a concatenation upper The by clothing. head to due outperforms margin head, large to compared informative less and Germany the of keepers goal respectively. the Argentina, are subjects two first dif- ( Romero of subjects performance three the on show approaches We ferent qualitative study dataset. a soccer a such the perform recognition, using We previously. in done not helps is evaluation clothing that obvious distance. a at people recognize to repre- better able develop are to that suggest sentations This approaches. the all for and informative extremely be head. to to compared found robust is face 12), Figure player. each of show performance also We recognition dataset. (bottom) Soccer the training of the split in (middle) test player and each (top) for images of Number 16: Figure sse nFgr 9 pe oyrgo,wihi often is which region, body upper 19, Figure in seen As clothing? is informative How poor is performance overall the dataset, soccer the On Recognition accuracy (%) #images in test clips #images in train clips 1000 2000 3000 4000 5000 1000 2000 3000 4000 100 20 40 60 80 and
Manuel Neuer
Referee Benedikt Hoewedes Mats Hummels Schweinsteiger Mesut Oezil Miroslav Klose
ihuiu ltigi iue1.The 19. Figure in clothing unique with ) Thomas Mueller Phillip Lahm Toni Kroos Jerome Boateng naeil Christoph Kramer Mario Gotze Andre Schurrle Sergio Romero Ezequiel Garay r identical. are Pabli Zabaleta huhi sintuitively is it Though Lucas Biglia aulNeuer Manuel Enzo Perez Gonzalo Huguain Lionel Messi Javier MAscherano Martin Demichelis Marcos Rojo Ezequiel Lavezzi Fernando Gago
, Sergio Aguero
Sergio Rodrigo Palacio Referee Soccer Clothing
Performance on 5 lead actors
100 Our approach (61.17%) 100 naeil (49.25%) Joint training (50.41%) Head-Upper body (49.03%) 80 80 Head (41.80%) Upper body (22.02%) Face (42.08%) 60 60
40 Accuracy(%) 40 20
20 0 Manuel Neuer (GER) Sergio Romero (ARG) Referee
0 Our approach (97.09%) naeil (97.09%) Diane Wiest Woody Allen Barbara Hershey Mia Farrow Michael Caine Joint training (95.53%) Head-Upper body (89.96%) Head (86.61%) Upper body (95.25%) Figure 17: Recognition performance of five lead actors in Hannah dataset. Figure 19: Effect of clothing on recognition. Performance on 5 lead players
40 Our approach (14.58%) naeil (12.40%) Joint training (10.16%) Head-Upper body (11.57%) 32 Head (12.54%) Upper body (8.53%)
24
Accuracy(%) 16
8
0 Schweinsteiger Javier Mascherano Lionel Messi Marcos Rojo Martin Demichelis
Figure 18: Recognition performance of five most occurring players in Soccer dataset. Figure 20: Recognition performance of Hannah movie set using IMDB plus samples from the Hannah test set. It is interesting to note that, convnets that are trained for identity recognition can distinguish clothing without any IV. Qualitative Results explicit modeling or hand-crafted features [29]. Confusion between identities: We show the recogni- We show some qualitative results in Figures 23 to 27. tion confusion matrix for Hannah and Soccer datasets in Figure 23 shows the success and failure cases of joint train- Figure 21 and Figure 22 respectively, with and without ing and separate training of body regions. We notice an tracking. We notice two important points related to gender over-influence of clothing while using separately trained and clothing. As seen from Figure 21, female subjects are and concatenated regional features, compared to the jointly mostly getting confused with female subjects, and similarly training features. In Figure 24, we show the effectiveness the male subjects are confused with male subjects. In Fig- of using multiple classifiers from each PSM. As seen in the figure, the concatenated head and upper body features ( ) ure 22, we notice that players from each team are mislabeled F with the members from the same team. These studies show may predict incorrect labels even when one (or two) of these the effectiveness of convnets in capturing human attributes features predict correctly, due to the over influence of less without any explicit training. Finally, majority voting over informative body region. Combining these three features is a track helps to produce consistent predictions. found to be more robust. Domain gap: To understand the effect of domain con- We show the top scoring predictions obtained from each trast between train and test instances, we conduct an ex- pose-specific PSM in Figure 25. It clearly shows how each periment adding different number of Hannah instances per PSM helps in the prediction of instances in that particular subject to the IMDB training gallery. The results are shown pose when the base model is unable to predict correctly. Fi- in Figure 20. As seen from the graph, the addition of even nally, we show the success and failure cases of our approach a few instances from the test domain results in a very large on Hannah and Soccer datasets in Figure 26 and Figure 27 improvement in the recognition performance. respectively, and compare with the naeil. iue21: Figure Christian Clemenson Christian Christian Clemenson Christian Maureen OSullivan Maureen Julia Louis-Dreyfus Julia Maureen OSullivan Maureen Julia Louis-Dreyfus Julia Verna O. Hobson Verna O. Hobson Barbara Hershey Barbara Barbara Hershey Barbara Joanna Gleason Joanna Joanna Gleason Joanna Max Max Von Sydow Richard Jenkins Richard Max Max Von Sydow Richard Jenkins Richard Sam Sam Waterston Sam Sam Waterston Fred Fred Melamed Fred Fred Melamed Michael Caine Michael Michael Caine Michael Ken Costigan Ken Ken Costigan Ken Tony Roberts Tony Roberts Carrie Fisher Carrie Carrie Fisher Carrie John John Turturo John John Turturo Woody Allen Daniel Stern Daniel Kavner Julie Woody Allen Bobby Short Bobby Daniel Stern Daniel Kavner Julie Bobby Short Bobby Diane Wiest Diane Diane Wiest Diane Lloyd Nolan Lloyd Lewis Black Lewis Lloyd Nolan Lloyd Lewis Black Lewis ofso matrix Confusion Mia Farrow Mia Mia Farrow Mia Paul Bates Paul Paul Bates Paul Barry Barry Gibb Barry Barry Gibb J.T. Walsh J.T. Walsh 12.99% 32.36% 15.62% 11.11% 13.56% 0.00% 3.17% 4.97% 0.84% 3.04% 1.92% 2.09% 7.19% 6.66% 0.00% 2.74% 3.16% 1.38% 0.75% 2.51% 5.07% 3.31% 7.54% 1.71% 8.67% 4.73% 0.48% 0.00% 0.79% 4.17% 0.49% 0.00% 1.08% 0.17% 9.88% 0.00% 0.00% 2.74% 7.59% 0.39% 0.00% 1.35% 3.00% 3.85% 6.07% 0.60% 3.86% 6.07% 2.78% 0.00% 0.00% 3.75% Carrie Fisher (F) Carrie Fisher (F) 24.95% 33.45% 39.46% 0.79% 0.80% 0.67% 1.52% 0.00% 0.00% 0.07% 0.00% 0.59% 0.00% 0.00% 0.00% 1.57% 0.00% 0.09% 5.89% 0.69% 6.01% 1.01% 1.56% 3.25% 0.00% 3.78% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.04% 4.64% 0.00% 7.65% 0.00% 1.59% 9.39% 0.00% 0.00% 0.00% 0.00% Joanna Gleason (F) Joanna Gleason (F) 100.00% 60.90% 15.73% 33.05% 49.70% 43.14% 28.06% 36.94% 15.21% 18.57% 23.04% 24.86% 77.49% 10.01% 49.70% 44.33% 59.80% 20.14% 56.25% 12.72% 31.07% 29.34% 82.32% 39.03% 52.85% 16.11% 21.22% 30.58% 84.29% 85.98% 71.79% 18.15% 1.58% 0.53% 2.01% 6.84% 6.97% 2.24% 8.23% 4.31% 4.52% 0.00% 0.00% 2.03% 3.80% 4.69% 0.00% 0.00% 5.50% 2.34% 2.04% 2.03% Julia Louis-Dreyfus (F) Julia Louis-Dreyfus (F) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Verna O. Hobson (F) Verna O. Hobson (F) 0.31% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.03% 0.24% 0.00% 0.00% 0.60% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Maureen Osullivan (F) Maureen Osullivan (F) 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Julie Kavner (F) Julie Kavner (F) nHna aae tp ihad(otm ihu tracking. without (bottom) and with (top) dataset Hannah on 79.29% 46.78% 88.12% 46.65% 0.00% 2.91% 1.76% 3.40% 0.38% 0.75% 0.00% 4.37% 0.00% 0.39% 0.00% 2.49% 0.00% 0.00% 0.25% 0.75% 0.00% 1.49% 8.45% 0.00% 0.00% 0.00% 1.18% 9.48% 0.00% 0.00% 4.01% 3.75% 0.00% 0.00% 0.00% 0.32% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.21% 0.00% 1.23% 5.69% 0.00% 0.00% 0.00% 0.00% 6.90% Mia Farrow (F) Mia Farrow (F) 21.96% 22.74% 16.40% 19.02% 0.00% 0.00% 1.63% 1.52% 0.09% 0.00% 2.58% 0.30% 1.42% 0.00% 0.00% 0.00% 0.98% 4.72% 0.34% 0.19% 0.52% 1.54% 0.50% 4.22% 0.10% 0.76% 2.13% 2.47% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 3.63% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 7.50% 0.00% 0.00% 0.08% 0.00% 0.00% 2.43% 0.00% 0.00% 0.00% 1.25% Diane Wiest (F) Diane Wiest (F) 57.90% 15.81% 32.75% 10.37% 12.85% 10.58% 13.66% 19.06% 62.75% 15.61% 28.21% 20.00% 6.33% 9.04% 5.65% 5.16% 3.63% 1.96% 4.18% 1.89% 2.39% 8.68% 4.76% 2.67% 8.75% 1.03% 4.80% 5.69% 0.00% 0.00% 3.24% 0.00% 0.00% 3.41% 4.51% 1.87% 3.40% 0.85% 2.68% 0.00% 0.00% 0.53% 0.00% 1.59% 9.32% 0.00% 0.00% 2.98% 0.00% 2.74% 0.00% 0.00% Barbara Hershey (F) Barbara Hershey (F) 42.32% 51.61% 49.30% 46.45% 0.00% 0.00% 0.00% 0.68% 0.00% 0.05% 0.06% 0.00% 0.00% 0.05% 0.00% 0.00% 0.05% 5.42% 0.00% 0.00% 4.64% 0.00% 0.26% 0.85% 1.28% 0.00% 0.05% 1.00% 0.00% 0.00% 0.00% 1.05% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.62% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Sam Waterston Sam Waterston 12.25% 73.04% 11.98% 21.11% 11.43% 82.05% 22.46% 0.00% 2.75% 3.95% 0.13% 0.77% 0.07% 4.10% 0.09% 1.74% 0.00% 0.03% 0.35% 0.00% 2.65% 0.00% 4.46% 0.00% 0.00% 0.00% 4.36% 0.00% 2.74% 0.00% 0.00% 0.00% 0.00% 0.94% 0.00% 6.86% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 4.12% 5.89% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Woody Allen Woody Allen 0.00% 0.00% 0.16% 0.05% 0.00% 0.00% 0.00% 0.02% 0.00% 0.05% 0.00% 0.00% 0.00% 0.00% 0.03% 0.02% 0.00% 1.12% 0.03% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Tony Roberts Tony Roberts 0.00% 0.26% 0.00% 0.00% 0.57% 0.47% 0.00% 0.89% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.01% 0.16% 0.00% 0.02% 0.02% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 2.40% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Lewis Black Lewis Black 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Paul Bates Paul Bates 74.31% 97.26% 0.00% 0.00% 0.00% 0.36% 0.00% 0.11% 0.06% 0.04% 0.00% 0.00% 0.10% 0.00% 0.00% 0.00% 0.10% 0.00% 0.00% 0.22% 0.00% 1.38% 0.00% 0.06% 0.00% 0.39% 0.00% 0.00% 0.00% 0.00% 0.28% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 3.08% 0.00% 0.00% 0.00% 0.00% 0.00% Fred Melamed Fred Melamed 22.49% 55.51% 10.97% 40.72% 20.32% 66.14% 75.86% 15.00% 60.78% 14.52% 0.00% 0.20% 0.00% 3.92% 0.24% 0.27% 0.10% 0.12% 0.04% 0.00% 0.10% 0.00% 0.00% 0.03% 1.08% 0.00% 6.10% 0.00% 1.26% 0.20% 7.48% 0.00% 0.00% 0.00% 3.61% 0.00% 0.08% 0.00% 0.02% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Richard Jenkins Richard Jenkins 44.08% 17.92% 52.42% 0.00% 5.30% 0.26% 1.17% 0.22% 0.66% 0.07% 0.11% 0.03% 0.35% 0.00% 0.00% 0.43% 3.06% 2.91% 2.40% 0.47% 4.18% 0.43% 0.00% 0.43% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.21% 0.00% 0.03% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 5.03% 0.00% 0.18% 0.00% 0.00% 0.00% 0.00% 0.00% 6.42% 0.00% 0.00% Daniel Stern Daniel Stern 0.00% 0.20% 0.00% 0.00% 0.00% 0.04% 0.00% 0.01% 0.14% 0.00% 0.15% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.05% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.25% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% J.T. Walsh J.T. Walsh 81.65% 27.22% 18.44% 45.24% 52.33% 49.77% 92.41% 10.54% 10.68% 32.17% 10.49% 15.54% 63.22% 56.44% 61.10% 4.72% 1.41% 6.21% 3.85% 9.95% 8.93% 7.03% 3.11% 4.77% 2.77% 0.56% 8.10% 8.73% 8.65% 3.04% 0.85% 0.30% 7.10% 0.00% 0.25% 0.98% 0.59% 5.25% 2.12% 5.50% 0.00% 5.27% 0.00% 0.00% 0.00% 0.53% 5.13% 0.00% 0.00% 3.28% 0.00% 0.00% Michael Caine Michael Caine 36.94% 15.09% 14.94% 47.46% 26.72% 22.56% 20.11% 55.37% 0.00% 1.44% 0.39% 3.28% 0.51% 0.07% 0.49% 7.65% 0.00% 0.00% 0.19% 1.78% 0.26% 1.12% 1.93% 1.90% 5.52% 2.61% 0.90% 3.04% 0.00% 1.50% 0.00% 0.35% 0.11% 3.73% 0.07% 0.00% 0.36% 3.28% 0.00% 0.00% 0.00% 0.00% 0.00% 0.96% 0.54% 0.00% 5.84% 2.24% 0.00% 5.24% 0.00% 0.00% John Turturo John Turturo 0.10% 0.00% 0.00% 0.02% 0.00% 2.18% 0.00% 0.03% 0.00% 0.05% 0.00% 1.50% 0.00% 0.00% 0.00% 0.01% 0.00% 0.00% 0.01% 0.00% 0.00% 0.00% 0.05% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 1.57% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Lloyd Nolan Lloyd Nolan 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Ken Costigan Ken Costigan 0.00% 5.29% 0.00% 7.98% 0.76% 1.40% 0.00% 0.37% 0.00% 0.05% 0.00% 0.00% 0.00% 0.00% 0.19% 0.21% 0.03% 0.21% 0.06% 0.19% 0.00% 0.04% 0.05% 0.00% 0.00% 0.03% 0.00% 0.00% 0.00% 6.44% 0.00% 0.22% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.03% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Max Von Sydow Max Von Sydow 19.07% 0.00% 0.00% 8.17% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.01% 0.00% 0.00% 0.03% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Barry Gibb Barry Gibb 23.28% 17.60% 10.58% 17.01% 0.59% 0.48% 0.00% 5.54% 0.00% 0.42% 0.90% 0.93% 0.00% 0.25% 0.63% 0.00% 0.37% 0.70% 0.00% 0.58% 0.35% 1.87% 0.04% 0.62% 0.45% 0.00% 0.20% 1.30% 0.00% 0.00% 0.00% 4.05% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.07% 0.00% 0.13% 0.01% 1.80% 0.00% 0.05% 0.00% 0.00% 0.00% 0.00% Christian Clemenson Christian Clemenson 100.00% 59.17% 0.00% 0.00% 0.03% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% Bobby Short Bobby Short iue22: Figure Benedikt Hoewedes Benedikt Benedikt Hoewedes Benedikt Javier MAscherano Javier Javier MAscherano Javier Martin Demichelis Martin Martin Demichelis Martin Christoph Kramer Christoph Christoph Kramer Christoph Gonzalo Huguain Gonzalo Gonzalo Huguain Gonzalo Ezequiel Lavezzi Ezequiel Ezequiel Lavezzi Ezequiel Jerome Boateng Jerome Jerome Boateng Jerome Thomas Mueller Thomas Thomas Mueller Thomas Rodrigo Palacio Rodrigo Fernando Gago Fernando Rodrigo Palacio Rodrigo Fernando Gago Fernando Ezequiel Garay Ezequiel Schweinsteiger Ezequiel Garay Ezequiel Sergio Romero Sergio Schweinsteiger Mats Hummels Andre Schurrle Andre Sergio Romero Sergio Andre Schurrle Andre Mats Hummels Miroslav Klose Miroslav Miroslav Klose Miroslav Pabli Zabaleta Pabli Sergio Sergio Aguero Pabli Zabaleta Pabli Manuel Neuer Manuel Sergio Sergio Aguero Manuel Neuer Manuel Marcos Rojo Marcos Lionel Messi Lionel Marcos Rojo Marcos Phillip Lahm Phillip Lionel Messi Lionel Mario Gotze Mario Lucas Biglia Lucas Phillip Lahm Phillip Mario Gotze Mario Lucas Biglia Lucas Mesut Mesut Oezil Mesut Mesut Oezil Enzo Perez Enzo Enzo Perez Enzo Toni Kroos Toni Kroos ofso matrix Confusion Referee Referee 93.31% 100% 0.03% 0.00% 0.08% 0.00% 0.51% 0.04% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.73% 0.08% 0.00% 0.37% 0.00% 0.78% 0.00% 2.09% 0.13% 0.00% 0.00% 0.00% 0.00% 0.07% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Manuel Neuer Manuel Neuer 10.24% 16.38% 18.50% 0.00% 0.00% 0.00% 0.12% 0.24% 1.17% 0.43% 0.43% 2.80% 0.00% 1.22% 0.00% 0.09% 0.00% 9.28% 0.00% 0.63% 4.62% 7.63% 8.61% 3.76% 6.31% 7.95% 2.13% 0.80% 20% 19% 16% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 0% 0% 6% 4% 0% 1% 0% 0% Benedikt Hoewedes Benedikt Hoewedes 24.47% 17.76% 13.53% 12.96% 13.34% 0.06% 0.00% 0.00% 0.00% 0.00% 0.00% 0.11% 0.00% 0.03% 0.00% 1.52% 0.00% 0.58% 0.00% 0.04% 0.21% 5.19% 0.27% 1.86% 0.31% 6.89% 0.37% 0.00% 10% 21% 13% 21% 48% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 9% 9% 0% 0% 0% Mats Hummels Mats Hummels 23.55% 22.71% 62.77% 10.76% 10.92% 10.33% 26.42% 8.71% 0.00% 0.00% 0.00% 0.33% 0.00% 0.00% 0.48% 2.22% 2.52% 0.25% 0.61% 0.00% 0.00% 2.04% 0.00% 1.03% 9.76% 5.33% 7.69% 3.76% 40% 90% 12% 35% 0% 0% 0% 0% 0% 0% 0% 2% 0% 0% 0% 0% 0% 0% 0% 7% 0% 6% 0% 9% 7% 1% 0% 0% Schweinsteiger Schweinsteiger 14.79% 18.41% 10.07% 0.10% 4.62% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 1.66% 8.10% 0.00% 0.00% 0.00% 0.00% 0.00% 2.77% 0.00% 0.27% 1.51% 9.54% 2.62% 1.77% 0.55% 23% 14% 10% 21% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 8% 0% 0% 0% 0% 0% 8% 0% 0% 0% 0% 0% 0% Mesut Oezil Mesut Oezil 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.03% 0.08% 0.00% 0.00% 0.17% 0.00% 0.09% 0.00% 0.00% 0.00% 0.00% 0.00% 0.06% 0.00% 0.12% 0.07% 0.02% 0.00% 0.31% 0.00% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Miroslav Klose Miroslav Klose 16.95% 39.69% 36.14% 15.73% 11.06% 11.55% 0.00% 0.38% 0.00% 1.42% 0.55% 2.91% 0.10% 0.16% 4.69% 0.38% 2.88% 2.37% 0.63% 0.00% 0.00% 8.82% 4.52% 8.66% 8.03% 2.97% 8.71% 0.00% 23% 83% 64% 33% 17% 16% 11% 17% 0% 0% 0% 0% 0% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 7% 7% 2% 6% 0%
nSce aae tp ihad(otm ihu tracking. without (bottom) and with (top) dataset Soccer on Thomas Mueller Thomas Mueller 12.83% 13.87% 18.56% 24.98% 19.36% 12.21% 18.50% 0.00% 0.00% 0.38% 0.00% 0.00% 0.00% 0.00% 0.98% 0.18% 0.00% 0.00% 0.00% 0.06% 0.00% 0.09% 0.04% 3.41% 6.49% 6.96% 1.72% 9.77% 13% 25% 13% 11% 13% 23% 33% 23% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 7% 0% 0% Phillip Lahm Phillip Lahm 39.93% 15.93% 10.04% 10.25% 0.25% 1.02% 0.46% 0.00% 0.00% 0.96% 0.33% 0.28% 8.10% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 5.76% 2.60% 2.17% 3.85% 6.56% 6.04% 8.87% 1.05% 0.00% 53% 20% 12% 10% 12% 0% 0% 0% 0% 0% 0% 0% 0% 9% 0% 0% 0% 0% 0% 0% 2% 0% 0% 0% 0% 8% 0% 0% Toni Kroos Toni Kroos 16.09% 16.87% 15.45% 11.32% 12.08% 0.00% 0.00% 2.45% 0.00% 3.67% 7.10% 4.53% 4.13% 4.80% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.40% 0.38% 0.00% 0.00% 0.63% 0.00% 0.33% 14% 31% 25% 14% 12% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 6% 0% 7% 3% 0% Jerome Boateng Jerome Boateng 0.00% 0.00% 0.13% 0.00% 0.00% 0.00% 0.07% 0.10% 0.19% 0.00% 0.00% 0.00% 0.04% 1.35% 0.00% 2.45% 0.00% 0.00% 0.71% 0.35% 1.04% 1.85% 0.00% 0.44% 0.77% 1.67% 0.18% 0.00% 0% 0% 0% 0% 0% 0% 0% 1% 1% 0% 0% 0% 4% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Christoph Kramer Christoph Kramer 13.42% 0.00% 0.04% 0.00% 0.00% 0.41% 0.13% 0.73% 0.17% 2.54% 6.56% 2.35% 0.21% 0.00% 0.00% 0.55% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 1.36% 0.00% 0.00% 0.00% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 6% 0% 0% 0% 0% Mario Gotze Mario Gotze 13.92% 0.00% 0.00% 4.58% 0.00% 0.00% 7.05% 4.88% 2.06% 4.73% 0.22% 0.00% 0.31% 1.05% 0.00% 0.00% 1.02% 0.00% 0.00% 0.00% 0.03% 0.69% 1.40% 0.00% 0.00% 0.00% 0.00% 0.00% 25% 0% 0% 0% 0% 0% 5% 5% 0% 7% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Andre Schurrle Andre Schurrle 92.29% 0.00% 0.26% 0.00% 0.00% 0.00% 0.03% 0.00% 0.99% 0.03% 0.83% 0.00% 0.32% 0.67% 0.00% 0.11% 0.00% 0.09% 0.00% 0.00% 0.00% 0.13% 0.00% 0.00% 1.34% 0.00% 0.00% 0.00% 95% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Sergio Romero Sergio Romero 28.15% 39.87% 32.50% 21.82% 16.64% 16.13% 0.00% 8.74% 0.00% 2.90% 5.06% 1.16% 1.22% 3.70% 5.90% 0.11% 4.38% 2.71% 0.00% 0.00% 0.00% 2.72% 2.29% 1.66% 6.95% 0.45% 8.34% 9.45% 28% 46% 32% 27% 16% 19% 0% 0% 0% 0% 0% 4% 0% 2% 5% 0% 0% 0% 3% 0% 0% 0% 8% 0% 0% 0% 0% 0% Ezequiel Garay Ezequiel Garay 15.27% 0.36% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.06% 0.00% 0.00% 0.00% 0.04% 0.00% 0.00% 0.00% 0.68% 0.51% 0.00% 0.00% 2.02% 0.11% 0.43% 0.00% 0.00% 0.00% 0.00% 3.33% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 6% Pabli Zabaleta Pabli Zabaleta 11.51% 0.00% 5.12% 0.53% 0.00% 2.14% 8.59% 4.64% 5.93% 6.96% 0.00% 5.12% 4.99% 7.05% 0.04% 0.00% 0.00% 0.27% 0.00% 0.00% 0.43% 0.29% 0.00% 0.29% 0.06% 1.41% 0.00% 0.30% 11% 10% 4% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 6% 0% 0% 1% 0% Lucas Biglia Lucas Biglia 24.68% 12.52% 23.35% 0.00% 0.00% 0.00% 0.09% 0.00% 0.00% 0.06% 0.21% 0.00% 0.00% 0.00% 0.00% 0.00% 1.22% 0.00% 2.82% 5.04% 0.00% 0.24% 8.03% 8.16% 1.30% 4.97% 1.73% 3.79% 25% 12% 10% 10% 17% 12% 0% 0% 0% 0% 0% 0% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Enzo Perez Enzo Perez 55.34% 23.66% 58.36% 21.31% 11.87% 14.07% 11.61% 47.99% 38.05% 28.49% 31.46% 19.98% 0.00% 7.68% 0.99% 0.00% 0.00% 2.17% 0.71% 0.00% 1.77% 6.26% 0.40% 0.07% 0.13% 3.65% 2.65% 3.04% 72% 68% 21% 15% 11% 61% 46% 28% 33% 21% 0% 0% 0% 0% 0% 0% 0% 8% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% Gonzalo Huguain Gonzalo Huguain 10.77% 12.52% 3.55% 0.52% 0.00% 0.00% 0.00% 1.66% 0.22% 0.00% 1.81% 0.12% 0.00% 0.00% 1.04% 0.00% 0.00% 0.06% 0.13% 3.32% 4.58% 3.20% 1.47% 5.46% 5.94% 1.51% 1.34% 0.46% 15% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 0% 9% 1% 0% Lionel Messi Lionel Messi 0.36% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.49% 0.04% 0.00% 0.00% 0.17% 0.00% 0.00% 0.00% 0.86% 0.00% 0.00% 0.00% 0.12% 1.37% 1.60% 2.98% 1.37% 5.45% 0.00% 1.41% 1.37% 22% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 5% 0% 0% 0% 0% Javier MAscherano Javier MAscherano 18.82% 0.00% 0.00% 0.00% 0.00% 0.10% 0.04% 0.25% 0.96% 0.00% 0.00% 0.00% 0.04% 0.22% 0.00% 0.00% 0.00% 0.00% 0.83% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 20% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 0% 0% 0% 0% Martin Demichelis Martin Demichelis 0.18% 0.00% 3.05% 0.00% 0.24% 8.91% 2.40% 1.17% 2.39% 0.00% 0.00% 2.94% 2.95% 2.87% 0.09% 0.00% 0.00% 0.00% 0.00% 0.17% 0.00% 0.38% 0.00% 3.54% 0.13% 0.00% 2.22% 0.00% 0% 0% 0% 0% 0% 8% 0% 0% 3% 0% 0% 6% 4% 8% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Marcos Rojo Marcos Rojo 0.00% 0.00% 0.00% 0.00% 0.12% 0.79% 1.49% 3.08% 2.39% 0.00% 0.00% 3.84% 0.00% 1.03% 0.00% 0.00% 0.00% 0.00% 0.06% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.25% 0.00% 0% 0% 0% 0% 0% 0% 0% 6% 2% 0% 0% 7% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Ezequiel Lavezzi Ezequiel Lavezzi 0.00% 0.00% 0.46% 0.00% 0.00% 0.00% 0.00% 0.00% 1.40% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.24% 0.00% 0.00% 0.00% 4.05% 0.00% 0.00% 0.00% 0.00% 0.00% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 9% 0% 0% 0% 0% 0% Fernando Gago Fernando Gago 16.32% 10.96% 10.40% 11.25% 0.62% 5.38% 0.00% 0.12% 2.15% 4.00% 0.08% 1.54% 4.79% 2.33% 0.22% 0.00% 0.00% 0.00% 1.07% 1.21% 0.73% 0.34% 0.76% 0.59% 2.48% 2.76% 0.00% 0.00% 20% 13% 10% 0% 0% 0% 0% 3% 0% 0% 9% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 3% 0% 0% Sergio Aguero Sergio Aguero 56.34% 12.21% 14.50% 17.69% 23.71% 45.53% 29.90% 32.39% 31.00% 16.63% 1.91% 1.42% 2.42% 0.00% 8.53% 4.39% 1.28% 9.52% 5.61% 0.00% 1.51% 7.45% 0.00% 2.98% 1.40% 0.05% 5.55% 0.00% 80% 17% 19% 35% 58% 40% 37% 52% 11% 12% 0% 8% 0% 0% 0% 5% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Rodrigo Palacio Rodrigo Palacio 91.93% 100% 0.00% 0.07% 0.00% 0.00% 0.55% 0.00% 0.46% 0.03% 0.00% 0.00% 0.06% 0.12% 0.04% 0.22% 0.11% 0.00% 0.00% 1.36% 0.22% 0.00% 0.00% 0.00% 0.00% 0.77% 3.03% 0.00% 0.00% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% Referee Referee JT ✅ JT ❌
ST ST ❌ ✅
JT ✅ JT ❌
ST ❌ ST ✅
JT ✅ JT ❌
ST ❌ ST ✅
Figure 23: Success and failure cases of separate and joint training of body regions on PIPA dataset. Column one shows the test images and the column two shows the training images belonging to the predicted subject. (Left) shows the success and failure case of joint training (JT) and separate training (ST), respectively and the reverse is shown in (right). ∑i ∑i ✅ ✅
❌ ❌ F F
✅ ✅ Fh Fh
❌ ❌ Fu Fu
∑i ✅ ∑i ✅
❌ ❌ F F
❌ ✅ Fh Fh
✅ ✅ Fu Fu
Figure 24: Effectiveness of multiple classifiers from each PSM: Column one shows the PIPA test images and the column two shows the training images belonging to the predicted subject using different approaches. The four approaches considered are the classifiers trained on head ( ) and upper body ( ) features, a classifier trained on concatenated head and upper body Fh Fu ( ) feature, and linear combination of three classifiers ( ) trained on these features. It clearly shows that it is advantageous F i to consider individual classifiers trained on regional features and their combination for improved performance. P base ❌ base + right ✅
base ❌ base + semi-right ✅
base ❌ base + frontal ✅
base ❌ base + semi-left ✅
base ❌ base + left ✅
Figure 25: Success cases of pose-specific models (PSMs) on PIPA dataset. Each row shows the success predictions of our approach where the improvement is obtained primarily due to the specific-pose model i.e., base model wrongly predicts but base + correct PSM predicts correctly. Green and yellow boxes indicate the success and failure result of naeil respectively. ours ours
naeil naeil
ours ours
naeil naeil
ours ours
naeil naeil
Figure 26: Comparison of our approach with naeil on Hannah dataset. Column one shows the test images and the column two shows the training images belonging to the predicted subject. (Left) in green shows the success case of our approach and the failure case of naeil. (Right) in red shows the failure case of our approach and the success case of naeil. ours ours
naeil naeil
ours ours
naeil naeil
ours ours
naeil naeil
Figure 27: Comparison of our approach with naeil on Soccer dataset. Column one shows the test images and the column two shows the training images belonging to the predicted subject. (Left) in green shows the success case of our approach and the failure case of naeil. (Right) in red shows the failure case of our approach and the success case of naeil.