Chalearn Multi-Modal Gesture Recognition
Total Page:16
File Type:pdf, Size:1020Kb
Human Pose Recovery and Behavior Analysis Group ChaLearn Looking at People Challenge 2015: Dataset and Results http://gesture.chalearn.org/ Xavier Baró, UOC & CVC Jordi Gonzàlez, UAB & CVC Junior Fabian, UAB & CVC Miguel Ángel Bautista, UB & CVC Marc Oliu, UB & CVC Hugo J. Escalante, INAOE & ChaLearn Isabelle Guyon, ChaLearn Sergio Escalera, UB & CVC & ChaLearn All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Group http://gesture.chalearn.org/ Challenge on multi-modal gesture recognition and cultural event recognition •Track on Action/Interaction Recognition: 235 performances of 11 action/interaction categories are recorded and manually labeled in continuous RGB sequences of different people performing natural isolated and collaborative behaviors. (Japan), among others. 2 Human Pose Recovery and Behavior Analysis Group http://gesture.chalearn.org/ Challenge on multi-modal gesture recognition and cultural event recognition •Track on Cultural Event Recognition: More than 10,000 images corresponding to 50 different cultural event categories will be considered. Examples of cultural events will be Carnival (Brasil, Italy, USA), Oktoberfest (Germany), San Fermin (Spain), Maha-Kumbh- Mela (India) and Aoi-Matsuri (Japan), among others. 3 Human Pose Recovery and Behavior Analysis Group •Track on Action/Interaction Recognition: 235 performances of 11 http://gesture.chalearn.org/ action/interaction categories are recorded and manually labeled in continuous RGB sequences of different people performing natural isolated and collaborative behaviors. • 235 action/interaction samples performed by 14 actors. • Large difference in length about the performed actions and interactions. • Several distracter actions out of the 11 categories are also present. • 11 action categories, containing isolated and collaborative actions: Wave, Point, Clap, Crouch, Jump, Walk, Run, Shake Hands, Hug, Kiss, Fight. There is a high intra-class variability among action samples. Overlap evaluation 4 Human Pose Recovery and Behavior Analysis Group •Track on Action/Interaction Recognition: 235 performances of 11 http://gesture.chalearn.org/ action/interaction categories are recorded and manually labeled in continuous RGB sequences of different people performing natural isolated and collaborative behaviors. Action categories Wave Point Clap Crouch Jump Walk Run Interaction categories Shake Hands Hug Kiss Fight 5 Human Pose Recovery and Behavior Analysis Group http://gesture.chalearn.org/ •Track on Cultural Event Recognition: More than 10,000 images corresponding to 50 different cultural event categories will be considered. • First dataset on cultural events •10.000 images corresponding to 50 cultural events. • Person related events. • High intra and inter-class variability. • Different cues can be exploited like garments, human poses, crowds analysis, objects and background scene. 6 Human Pose Recovery and Behavior Analysis Group http://gesture.chalearn.org/ •Track on Cultural Event Recognition: More than 10,000 images corresponding to 50 different cultural event categories will be considered. Inter-class variability 7 Human Pose Recovery and Behavior Analysis Group http://gesture.chalearn.org/ •Track on Cultural Event Recognition: More than 10,000 images corresponding to 50 different cultural event categories will be considered. Inter-class variability Carnival of Dunkerque Carnival of Rio Carnival of Venice Carnival of Helsinki Nothing Hill Carnival Carnival of Quebec 8 Human Pose Recovery and Behavior Analysis Group http://gesture.chalearn.org/ •Track on Cultural Event Recognition: More than 10,000 images corresponding to 50 different cultural event categories will be considered. Inter-class variability Quebec Winter Carnival Harbin Ice and Snow Festival 9 Cultural Event Country #Images 1. Annual Buffalo Roundup USA 334 2. Ati-atihan Philippines 357 3. Ballon Fiesta USA 382 4. Basel Fasnacht Switzerland 310 5. Boston Marathon USA 271 6. Bud Billiken USA 335 Human Pose Recovery and Behavior Analysis Group 7. Buenos Aires Tango Festival Argentina 261 8. Carnival of Dunkerque France 389 9. Carnival of Venice Italy 455 10. Carnival of Rio Brazil 419 11. Castellers Spain 536 12. Chinese New Year China http://gesture.chalearn.org/296 13. Correfocs Catalonia 551 •Track on Cultural Event Recognition: More 14.thanDesert Festi10val,of000Jaisalmer imagesIndia 298 15. Desfile de Silleteros Colombia 286 corresponding to 50 different cultural event categories will16. D´ıa bede los consideredMuertos . Mexico 298 Cultural Event Country #Images 17. Diada de Sant Jordi Catalonia 299 18. Diwali Festival of Lights India 361 1. Annual Buffalo Roundup USA 334 19. Falles Spain 649 2. Ati-atihan Philippines 357 20. Festa del Renaixement Tortosa Catalonia 299 3. Ballon Fiesta USA 382 21. Festival de laMarinera Peru 478 4. Basel Fasnacht Switzerland 310 22. Festival of the Sun Peru 514 5. Boston Marathon USA 271 23. Fiesta de laCandelaria Peru 300 6. Bud Billiken USA 335 24. Gion matsuri Japan 282 7. Buenos Aires Tango Festival Argentina 261 25. Harbin Ice and Snow Festival China 415 8. Carnival of Dunkerque France 389 26. Heiva Tahiti 286 9. Carnival of Venice Italy 455 27. Helsinki Samba Carnival Finland 257 10. Carnival of Rio Brazil 419 28. Holi Festival India 553 11. Castellers Spain 536 29. Infiorata di Genzano Italy 354 12. Chinese New Year China 296 30. LaTomatina Spain 349 13. Correfocs Catalonia 551 31. Lewes Bonfire England 267 Figure14. Desert3. CulturalFestival eofventsJaisalmersample images.India 298 15. Desfile de Silleteros Colombia 286 32. Macys Thanksgiving USA 335 16. D´ıa de los Muertos Mexico 298 33. Maslenitsa Russia 271 2011-12, the number17. Diadaofde SantimagesJordi is similar,Cataloniaaround 11, 000299 , 34. Midsommar Sweden 323 but the number18.ofDiwcateali goriesFestival ofisLightshere increasedIndiamore than361 5 35. Notting hill carnival England 383 36. Obon Festival Japan 304 times. 19. Falles Spain 649 20. Festa del Renaixement Tortosa Catalonia 299 37. Oktoberfest Germany 509 38. Onbashira Festival Japan 247 4. Pr otocol and21. Festievvalaluationde laMarinera Peru 478 22. Festival of the Sun Peru 514 39. Pingxi Lantern Festival Taiwan 253 This section23.introducesFiesta de laCandelariathe protocol and ePeruvaluation met-300 40. Pushkar Camel Festival India 433 41. Quebec Winter Carnival Canada 329 rics for both tracks.24. Gion matsuri Japan 282 25. Harbin Ice and Snow Festival China 415 42. Queens Day Netherlands 316 4.1. Evaluation26. Heiprva ocedure for action/interactionTahiti 286 43. Rath Yatra India 369 track 27. Helsinki Samba Carnival Finland 257 44. SandFest USA 237 28. Holi Festival India 553 45. San Fermin Spain 418 To evaluate29.theInfiorataaccuracdi Genzanoy of action/interactionItaly recogni-354 46. Songkran Water Festival Thailand 398 tion, we use the30.JaccardLaTomatinaIndex, the higher theSpainbetter. Thus,349 47. St Patrick’s Day Ireland 320 31. Lewes Bonfire England 267 48. The Battle of the Oranges Italy 276 Figure 3. Cultural events sample images. for then action and interaction categorieslabeled for aRGB sequence s, the32.JaccardMacys ThanksgiIndexvingisdefined as: USA 335 49. Timkat Ethiopia 425 33. Maslenitsa Russia 271 50. Viking Festival Norway 262 10 34. Midsommar T Sweden 323 Table 3. List of the 50 Cultural Events. 2011-12, the number of images is similar, around 11, 000, As,n B s,n 35. NottingJ hill= carnival S , England 383(1) but the number of categories is here increased more than 5 s,n 36. Obon FestivalAs,n B s,n Japan 304 times. where As,n isthe37. Oktoberfestground truth of action/interactionGermany n at509se- Jaccard Index among all categories for all sequences, where 38. Onbashira Festival Japan 247 quence s, and B s,n istheprediction for such an action at se- motion categories are independent but not mutually exclu- 4. Pr otocol and evaluation 39. Pingxi Lantern Festival Taiwan 253 quence s. As,n and B s,n are binary vectors where 1-values si ve (in a certain frame more than one action, interaction, This section introduces the protocol and evaluation met-correspond to frames40. PushkarinCamelwhichFestitheval n− th actionIndiaisbeing 433per- gesture class can be active). 41. Quebec Winter Carnival Canada 329 rics for both tracks. formed. The participants were evaluated based on the mean In the case of false positives (e.g. inferring an action 42. Queens Day Netherlands 316 4.1. Evaluation procedure for action/interaction 43. Rath Yatra India 369 track 44. SandFest USA 237 45. San Fermin Spain 418 To evaluate the accuracy of action/interaction recogni- 46. Songkran Water Festival Thailand 398 tion, we use the Jaccard Index, the higher the better. Thus, 47. St Patrick’s Day Ireland 320 for then action and interaction categorieslabeled for aRGB 48. The Battle of the Oranges Italy 276 sequence s, the Jaccard Index isdefined as: 49. Timkat Ethiopia 425 50. Viking Festival Norway 262 T Table 3. List of the 50 Cultural Events. A s,n B s,n Js,n = S , (1) A s,n B s,n where As,n isthe ground truth of action/interaction n at se- Jaccard Index among all categories for all sequences, where quence s, and B s,n istheprediction for such an action at se- motion categories are independent but not mutually exclu- quence s. A s,n and B s,n are binary vectors where 1-values si ve (in a certain frame more than one action, interaction, correspond to frames inwhich the n− th action isbeing per- gesture class can be active). formed. The participants were evaluated based on the mean In the case of false positives (e.g. inferring