sign in sign up
<<
Home , Bruce Grounds, Daihatsu Charmant, GMC Denali, Hiram Bithorn Stadium, Kia Venga, King Snake

ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

000 054 001 055 002 056 003 Supplementary Material: 057 004 058 005 Webly Supervised Learning of Convolutional Networks 059 006 060 007 061 008 Anonymous ICCV submission 062 009 063 010 Paper ID 465 064 011 065 012 066 013 067 014 In the supplementary material we include: Somewhat to our surprise, we found the extra clean-up 068 step does not help improving the detection performance. In 015 1. Additional results on PASCAL VOC for ablation anal- 069 fact, the average precision dropped for most of the cate- 016 ysis. 070 017 gories, regardless of whether the bounding box information 071 018 2. Scene classification results. is used or not. We suspect one reason lies in the size of 072 019 the data: after the object localization step, the number of 073 3. Diagnosis results for webly supervised object detec- 020 images was cut in more than a half (∼0.67M compared to 074 tion using [4]. 021 ∼1.5M). Also better algorithms can be devised for cleaning 075 022 4. Lists of objects, scenes and attributes. up web images. 076 023 On the other hand, fine-tuning ImageNet pretrained 077 024 1. Additional Results on PASCAL VOC model to Google images gives slightly better result than 078 training from scratch. However, further investigation is still 025 For ablation analysis, we provide more results on the 079 needed here since IN-GoogleA has seen more images (Im- 026 PASCAL VOC 2007 detection challenge. Please refer to 080 ageNet 1M) than GoogleA. 027 Table1 for comparison. The newly added results are shown 081 028 at the bottom rows. Following the notation in the paper, 082 029 “NFT” means before fine-tuning, and “FT” means after 2. Scene Classification 083 030 fine-tuning (100K iterations with a step size of 20K). We 084 To further demonstrate the usage of CNN features di- 031 add three pairs of new results: 085 032 rectly learned from the web, we also conducted a set 086 033 GoogleO-CI “CI” stands for “Cleaned Images”. This net- of scene classification experiments on the MIT Indoor-67 087 034 work is obtained by fine-tuning GoogleO on the im- dataset [5]. For each image, we simply computed the fc7 088 035 ages obtained by our object localization algorithm (de- feature vector, which has 4096 dimensions. We did not use 089 036 scribed in Section 3.3). Here, the entire image is re- any data augmentation or spatial pooling technique, with 090 037 garded as clean if at least one object is found inside the only pre-processing step normalizing the feature vector 091 038 it. However, the extra location information (bounding to unit `2 length [6]. The default SVM parameters (C=1) 092 039 box) is not used - the input is still the full image. were fixed throughout the experiments. 093 Table2 summarizes the results on the default train/test 040 GoogleO-CB “CB” stands for “Cleaned images with 094 split. We can see our web based CNNs achieved very com- 041 Bounding boxes”. Similar to GoogleO-CI, the net- 095 petetive performances: all the three networks achieved an 042 work is fine-tuned on the cleaned images. However, 096 accuracy at least on par with ImageNet pretrained mod- 043 instead of using the entire image, the image patches 097 els. Fine-tuning on hard images enhanced the features, but 044 cropped by the discovered bounding boxes are fed into 098 adding scene-related categories gave a huge boost to 66.5 045 the network. Both GoogleO-CI and Google-CB were 099 (comparable to the CNN trained on Places database [9], 046 fine-tuned for 200K iterations, with the learning rate 100 68.2). This indicates CNN features learned directly from 047 reduced every 40K iterations. 101 048 the web are indeed generic. 102 049 IN-GoogleA To see if more data can help the performance, Moreover, since we can get semantic labels (e.g. actions) 103 050 we fine-tuned the ImageNet pretrained network [3] to other than objects or scenes from the web for free, webly 104 051 all the images downloaded from Google. The fine- supervised CNN bears a great potential to perform well on 105 052 tuning was performed for 400K iterations, reducing many relevant tasks - and the cost is just as low as providing 106 053 learning rate every 80K iterations. a category list to query for that domain. 107

1 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

108 162 VOC 2007 test aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mAP 109 163 ImageNet-NFT [3] 57.6 57.9 38.5 31.8 23.7 51.2 58.9 51.4 20.0 50.5 40.9 46.0 51.6 55.9 43.3 23.3 48.1 35.3 51.0 57.4 44.7 110 164 111 GoogleO-NFT 57.1 59.9 35.4 30.5 21.9 53.9 59.5 40.7 18.6 43.3 37.5 41.9 49.6 57.7 38.4 22.8 45.2 37.1 48.0 54.5 42.7 165 112 GoogleA-NFT 54.9 58.2 35.7 30.7 22.0 54.5 59.9 44.7 19.9 41.0 34.5 40.1 46.8 56.2 40.0 22.2 45.8 36.3 47.5 54.2 42.3 166 Flickr-NFT 55.3 61.9 39.1 29.5 24.8 55.1 62.7 43.5 22.7 49.3 36.6 42.7 48.9 59.7 41.2 25.4 47.7 41.9 48.8 56.8 44.7 113 167 114 VOC-Scratch [1] 49.9 60.6 24.7 23.7 20.3 52.5 64.8 32.9 20.4 43.5 34.2 29.9 49.0 60.4 47.5 28.0 42.3 28.6 51.2 50.0 40.7 168 ImageNet-FT [3] 64.2 69.7 50.0 41.9 32.0 62.6 71.0 60.7 32.7 58.5 46.5 56.1 60.6 66.8 54.2 31.5 52.8 48.9 57.9 64.7 54.2 115 169 116 GoogleO-FT 65.0 68.1 45.2 37.0 29.6 65.4 73.8 54.0 30.4 57.8 48.7 51.9 64.1 64.7 54.0 32.0 54.9 44.5 57.0 64.0 53.1 170 GoogleA-NFT 64.2 68.3 42.7 38.7 26.5 65.1 72.4 50.7 28.5 60.9 48.8 51.2 60.2 65.5 54.5 31.1 50.5 48.5 56.3 60.3 52.3 117 Flickr-FT 63.7 68.5 46.2 36.4 30.2 68.4 73.9 56.9 31.4 59.1 46.7 52.4 61.5 69.2 53.6 31.6 53.8 44.5 58.1 59.6 53.3 171 118 172 GoogleO-CB-NFT 56.8 56.5 33.8 27.5 22.7 51.3 55.1 41.0 15.6 42.3 31.0 37.7 46.8 55.2 38.8 23.0 41.5 30.4 48.1 51.2 40.3 119 173 GoogleO-CB-FT 67.2 69.7 45.9 36.5 31.7 64.9 73.9 54.2 30.4 59.9 43.6 49.6 59.1 66.3 52.7 30.6 52.3 42.5 59.6 62.4 52.7 120 174 GoogleO-CI-NFT 55.7 57.1 33.6 27.4 22.6 55.6 57.3 39.3 17.6 39.7 35.0 35.5 46.9 57.0 39.5 21.0 43.1 31.1 49.4 52.1 40.8 121 175 GoogleO-CI-FT 63.2 67.0 41.2 34.7 30.7 66.7 73.1 56.5 30.2 58.0 44.6 50.4 57.1 67.4 52.6 31.2 53.1 41.9 60.1 61.9 52.1 122 176 IN-GoogleA-NFT 55.0 60.2 39.5 33.4 23.8 58.7 60.6 48.6 23.2 44.9 35.8 44.1 50.3 60.1 42.9 23.9 49.0 39.8 49.7 54.4 44.9 123 177 IN-GoogleA-FT 64.6 69.7 44.1 34.8 28.4 66.1 72.1 56.2 30.7 57.0 45.8 50.4 61.9 66.5 54.8 31.7 58.0 43.0 56.8 62.9 52.8 124 178 Table 1. Additional Results on VOC-2007 (PASCAL data used). 125 179

126 animals furniture person vehicles 180 100 100 100 100 127 Loc Loc Loc Loc 181 90 Sim 90 Sim 90 Sim 90 Sim Oth Oth Oth Oth 128 80 BG 80 BG 80 BG 80 BG 182 129 70 70 70 70 183 130 60 60 60 60 184 131 50 50 50 50 185 40 40 40 40 132 186 30 30 30 30 percentage of each type percentage of each type percentage of each type percentage of each type 133 20 20 20 20 187

134 10 10 10 10 188

0 0 0 0 135 25 50 100 200 400 800 1600 3200 25 50 100 200 400 800 1600 3200 25 50 100 200 400 800 1600 3200 25 50 100 200 400 800 1600 3200 189 136 total false positives total false positives total false positives total false positives 190 Figure 1. Diagnosis analysis using [4] for better understanding of the failure modes of our webly supervised pipeline. Please see top false 137 191 positives in Figure2 and3. 138 192 139 193 Indoor-67 Accuracy location information, the background is inevitably included 140 194 when the detector is trained (e.g. aeroplane, dining table). 141 ImageNet [9] 56.8 195 OverFeat [6] 58.4 Moreover, multiple objects can occur in the image, and the 142 196 algorithm has no clue that they should be treated as sepa- 143 GoogleO 58.1 197 rate pieces (e.g. bottle). Since our CNN representation is 144 GoogleA 66.5 198 Flickr 59.2 directly trained on full images, it is also learned to be in- 145 199 variant to such variations. All these factors caused serious 146 Table 2. Scene Classification Results on MIT Indoor-67 Dataset. 200 localization issues. 147 201 148 3. Diagnosis for Webly Supervised Object De- Second, we did observe some interesting semantic drift 202 149 tection between PASCAL categories and Google categories. For 203 150 example, bicycle can also mean motorcycle on Google. 204 151 The final analysis comes from the nice tool developed Careful disambiguation of different senses for a polysemous 205 152 in [4]. We would like to see what is the failure modes of word is needed here. Also note that our person detector 206 153 our webly supervised object detection (when no PASCAL is severely confused with cars, we suspect it is because 207 154 VOC train data is present). We took the results from our “caprice” was added as a related category and it can also 208 155 best model (Flickr-C) and fed them for diagnosis. Figure1, mean a car (chevy caprice). How to handle such issues is a 209 156 2 and3 highlight some of the interesting observations we fascinating future research topic by itself. 210 157 found. Hopefully these findings can offer more insights Finally, there is massive confusion within animal cate- 211 158 about how the web based detectors are performing and what gories (e.g. horse vs. cow). We believe it is mainly caused 212 159 are the possible directions to go for better performance. by the domain shifting between Google Images and PAS- 213 160 Firstly, localization error accounts for a lot of false pos- CAL images. Adding Flickr data can benefit this case (only 214 161 itives. Since Google Image Search results do not contain Google images are used to train our detectors). 215

2 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

216 270 217 271 218 272 219 273 220 274 221 275 222 aeroplane 276 223 277 224 278 225 279 226 280 227 281 228 282 229 bicycle 283 230 284 231 285 232 286 233 287 234 288 235 289 236 290 bottle 237 291 238 292 239 293 240 294 241 295 242 296 243 297 244 cat 298 245 Figure 2. Top false positives for selected categories on PASCAL VOC 2007 detection with Flickr-C. From top down: aeroplane, bicycle, 299 246 bottle, cat. 300 247 4. Lists of Concepts air conditioner, aircraft, aircraft carrier, airedale, air hockey, airliner, air- 301 248 plane, airplane nose, airship, airsoft gun, aisle, ak 47, akg, alarm, albacore, 302 249 4.1. Lists for Network Training albatross, albert einstein, ale, alfa alfetta, alfa romeo, alfa romeo quadri- 303 250 304 We obtained three lists of categories from ImageNet foglio, alfredo, alice cooper, alienware x17, allegra versace, alligator, al- 251 305 Challenge [7], SUN database [8] and NEIL knowledge ligator gar, alligator lizard, , , alp, alpaca, altar, alto 252 306 base [2]. saxophone, amber, ambulance, amc, american airlines, american alliga- 253 tor, american black bear, american chameleon, american coot, american 307 ImageNet syn-sets are transformed to its surface forms 254 egret, american limousine, american lobster, american organ, american 308 by just taking the first explanation. Most of them are fo- 255 rambler, american staffordshire terrier, amphibian, ana ivanovic, analog 309 cusing on object categories. To better assist querying and 256 clock, anchor, andre agassi, andrew garfield, android, anemone fish, an- 310 reducing noise, we remove the suffix (usually correspond 257 gel, angelfish, angkor, angora, ant, antelope, antlers, apartment, apiary, 311 to attributes, e.g. indoor/outdoor) of the SUN categories. 258 appenzeller, apple, apple chip, apple dumpling, apple fruit, apron, arabian 312 These two lists are augmented by the categories listed on 259 camel, arch shape, arctic fox, , armchair, arroz con pollo, artesia, 313 NEIL knowledge base, which are better suited as query 260 arthropod crab, artichoke, articulated bus, ashcan, asian elephant, aspara- 314 words. We also used NEIL’s list of attributes. 261 gus, asparagus soup, assault rifle, aston martin, aston martin db9, aston 315 In total we have 2,240 objects and 89 attributes to train 262 martin zagato, astrovette, atom bomb, atos, atv trailer, audi a6, audi a8, 316 the GoogleO net focusing on objects. 263 audi avant, australian terrier, autobot, avant, avenger, avocado, awl, axe, 317 264 1950 car, abacus, abaya, aberdonia, abs, academic gown, ac car, accor- axolotl, baboon, baby, bachelor, backboard, backgammon, backhoe, back- 318 265 dion, achiridae, acorn, acorn squash, acoustic guitar, acropolis, actor, ac- pack, bacon, badger, badminton, bagel, bagpipe, bagpipe chanter, bagre, 319 266 tress, acura, acura mdx, adidas ball, admiral, aeron, affenpinscher, afghan baguette, bakery, balance beam, bald eagle, balloon, ballplayer, ballpoint, 320 267 hound, african chameleon, african crocodile, african elephant, african grey, bamboo pipe, banana, banana boat, banana cream, band aid, banded gecko, 321 268 african grey parrot, african hunting dog, agama, agaric, agile, agra fort, banjo, banjolele, bannister, banyan, barack obama, barbell, barber chair, 322 269 aholehole, airbus 330, airbus 340, airbus a320, airbus a330, airbus a380, 323

3 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

324 378 325 379 326 380 327 381 328 382 329 383 330 384 dinning table 331 385 332 386 333 387 334 388 335 389 336 390 337 391 338 horse 392 339 393 340 394 341 395 342 396 343 397 344 398 345 person 399 346 400 347 401 348 402 349 403 350 404 351 405 352 tv monitor 406 353 Figure 3. Top false positives for selected categories on PASCAL VOC 2007 detection with Flickr-C. From top down: dining table, horse, 407 354 person, tv monitor. 408 355 409 barbershop, barge, baritone horn, barn, barndoor skate, barn spider, barom- bourdon, bouvier des flandres, bow, bowl, bowling ball, bow tie, boxer, 356 410 eter, barracouta, barracuda, barrel, barrow, baseball, baseball bat, basenji, box turtle, boy, bra, brabancon griffon, bracelet, brain coral, brambling, 357 411 basketball, basketball net, basking shark, bass, bass clarinet, basset, bass brass, brass choir, brassiere, brassy minnow, bread, breakwater, bream, 358 412 harmonica, bassinet, bassoon, bass sax, bat, bathing cap, bath towel, bath- breastplate, briard, brick, brick texture, bridge, bristol blenheim, brittany 359 413 tub, , baya fruta, baya weaver, bayon temple, bb gun, beach wagon, spaniel, broadbill, broadbill fish, broccoli, bronco nagurski, broom, broth, 360 414 beacon, beagle, beak, beaker, bean, bear, bearskin, beaver, bed, bedford, brown, brown bear, bruce banner, brussel sprout, bsa, bubble, bucket, buck- 361 415 bedlington terrier, bee, bee eater, stroganoff, beer, beer bottle, beer eye, buckle, , buick enclave, buick encore, building, bulbul, bull, 362 416 glass, beignet, belair, bell, bellagio fountain, bell cote, bell pepper, bench, bulletproof vest, bullet train, bullfrog, bull mastiff, burger, burrito, bus, 363 417 benq gh600, bentley, bentley gt, bernese, bernese mountain dog, beyerdy- bustard, butcher shop, butterfly, butterflyfish, butterfly perch, butternut 364 418 namic, bib, bible, bicycle, bicycle built for two, bicycle rack, big ben, big- squash, c3po, cab, cabbage, cabbage butterfly, cabriolet, , cadil- 365 419 eye thresher, bighorn, bikini, bill gates, binder, binoculars, biplane, bird, lac cts, cadillac elmiraj, cairn, calculator, caldron, california golden trout, 366 420 birdhouse, bison, bittern, black, black and gold garden spider, black and camaro, camera, camionnette, camry hybrid, candied yam, candle, can- 367 421 tan coonhound, black bream, black footed ferret, black grouse, blackmoor, nelloni, cannon, cannondale, canoe, can opener, cap, capital bikeshare, 368 422 black rice, black snook, black stork, black swan, black widow, bladefish, cappuccino, caprice, capuchin, car, caramel corn, carbonara, cardigan, car- 369 423 blenheim spaniel, bleu cheese dressing, blind shark, blizzard, bloodhound, doon, carisma, car mirror, carousel, carpenter kit, carrot puree, carrot raisin 370 424 blouse, blt, blue, blue acara, bluefish, blue groper, bluetick, bmw 320, salad, cart, carton, car wheel, cash machine, casio edifice, cassette, cas- 371 425 bmw d3, bmw isetta, bmw m67, bmx, boa , boar, boathouse, sette player, castle, cat, catamaran, caterpillar equipment, cathedral, cathe- 372 426 bobsled, boeing dreamliner, bolero, bolete, bolo tie, bolwell, bomb, bone dral of santiago de compostela, , cauliflower, cavefish, cd, cd 373 427 flute, bonfire, bongo drum, bonnet, booby, book, bookcase, book jacket, player, celery, cello, cellphone, cellular telephone, centipede, cereal bar, 374 428 bookshelf, bookshop, boombox, boomerang, border collie, border terrier, cessna aircraft, chai, chain, chainlink fence, chain mail, chainsaw, chain 375 429 borscht, borzoi, bosch, bosch product, bosch tools, bose, bose qc15, bossa saw, chain texture, challah, chambered nautilus, chamomile, chana dal, 376 430 nova, boston bull, boston celtics, boston terrier, bottlecap, bouldering, chandelier, changgo, , charlie chaplin, charlie hunnam, check tex- 377 431

4 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

432 486 ture, cheese, cheeseburger, cheese fondue, cheetah, cherokee 4wd, chero- campagnola, fiat dino, fiat doblo, fiat siena albea, fiddler crab, fig, fighter 433 487 kee rose, cherry, cherry pie, chery cross eastar, chesapeake bay retriever, plane, figure skater, file, finger, fingerling , fipple flute, fire, fireboat, 434 488 chess, chest, chevelle, chevrolet denali, chevrolet epica, chevro- fire engine, fire extinguisher, fire screen, fish, flag, flagpole, flamingo, flank 435 489 let , chevrolet panther, chevrolet syclone, chevy nova, chicago steak, flash bulb, flatbed, flat coated retriever, flatworm, fleetwood, flower, 436 490 bulls, chickadee, chiffonier, chihuahua, chime, chimpanzee, china cabi- flowerpot, flute, fly, flyingfish, flying squirrel, foggy weather, folding chair, 437 491 net, chisel, chiton, chocolate, chocolate sauce, chopsticks, chow, christ- foliage, food truck, football, football , ford , ford windstar, 438 492 mas stocking, christ redeemer, chromatic harmonica, chrysler concorde, fork, forkbeard, forklift, fountain, fountain pen, four poster, four string 439 493 chrysler jeep, chrysler vision, chubby, church, cicada, cinderella, cinema, banjo, fox, fox squirrel, fpv, fpv rc, freezer, freight car, french bulldog, 440 494 cinnamon stick, circular saw, citroen, citroen picasso, citroen projet, cit- french horn, french loaf, frida kahlo, fried egg, fried onion, frijoles, frilled 441 495 roen saxo, citroen sm, clam, clawhammer, cleaver, cliff, cliff dwelling, lizard, frisbee, frog, fruehauf, fruit desserts, fryer, frying pan, fur coat, 442 496 cloak, clock, clog, closet, cloud, clouds, cloudy weather, clownfish, clown furry texture, fusion hybrid, gar, garage, garbage truck, garden spider, gar- 443 497 knife, clubman, club wagon, clumber, cluttered, c max, coach, coach bus, lic, garlic soup, garter snake, gasmask, gas pump, gateway arch, gazelle, 444 498 coach purse, coat, cock, cocker spaniel, cockroach, cocktail shaker, co- gene bangert, george lucas, german shepherd, german short haired pointer, 445 499 conut rice, maker, coffee mug, coffeepot, coho, coil, cold scenery, geyser, giant defy, giant panda, giant schnauzer, gibbon, gila monster, gi- 446 500 collie, colobus, colosseum, combination lock, comet, comic book, com- raffe, girl, glasses, glove, glowlight tetra, denali, post, goat, gob- 447 501 mode, commodore sv6, common iguana, common newt, computer - let, goggle eye fish, go kart, gold, golden gate, golden retriever, goldfinch, 448 502 board, computer mouse, concert percussion, conch, cone shape, confec- goldfish, golf ball, golfcart, golf putter, golf trolley, gondola, gong, goonch, 449 503 tionery, conga drum, congrio, congrio dorado, consomme, container ship, goose, gordon setter, gorilla, gourd, gown, grado, grandeur, grand pi- 450 504 convertible, cooker, cooktop, cooper, coral fungus, coral grouper, coral ano, granny smith, grape, grasshopper, grass texture, gray, graysby, great 451 505 reef, cordless drill, cordless phone, corkscrew, corn, cornet, corolla, corsa, dane, greater swiss mountain dog, great grey owl, great pyrenees, great 452 506 corvette, coucal, couch, cougar, countryman, coupe, cow, cowboy, cow- white shark, green, green chili, greenhouse, green lizard, green mamba, 453 507 boy boot, cowboy hat, cowpea, coyote, cpu, cracker, cradle, crane, crane green snake, grenache, gretsch, grey fox, greyhound, greyhound bus, grey 454 508 fly, crash cymbal, crash helmet, crate, crayfish, crib, cricket, cricket ball, whale, grigio telesto, grille, grocery store, groenendael, groom, ground 455 509 crock pot, crocodile, croissant, crooked, croquet ball, , crossword beetle, guacamole, guenon, guillotine, guinea pig, guitar, gulf sturgeon, 456 510 puzzle, crowded, crumpet, crutch, cuatros, cube shape, cucumber, cuirass, gumpert apollo, gypsy moth, gyromitra, hacksaw, hair, hair blower, hair- 457 511 cup, cupboard, cupcake, curly coated retriever, curtain, custard apple, cyan, brush, hair dryer, hair slide, hair spray, half track, hamburger patty, ham- 458 512 cylinder shape, dacia, dacia logan, daewoo, daewoo prince, dai- burger soup, ham hock, hammer, hammered dulcimer, hammerhead, ham- 459 513 hatsu, charmant, daihatsu duster, daimler, daimler db18, daim- per, hamster, hand, handbag, hand blender, hand blower, handgun, hand 460 514 ler superv8, daisy, dalmatian, dam, damselfish, damselfly, dandie din- held computer, handkerchief, handrail, handycam, hard boiled egg, hard 461 515 mont, dandylion, dan tam, daph, dark , darth vader, , dat- disc, hard disk, hardyhead, hare, harmonica, harp, hartebeest, harvester, 462 516 sun maxima, datsun pickup, datsun pulsar, datsun sentra, datsun stanza, harvestman, hat, , hatchet, hay, head cabbage, headlight, head- 463 517 david beckham, decanter, decepticon, deer, delta, delta airlines, dentist phone, headset, helicopter, helmet, hen, hen of the woods, hermit crab, 464 518 chair, desert eagle, desk, desktop computer, deville, dhokla, dhol, dhole, hero cycles, hifiman, high heel, hiker, hilsa, hino ranger, hip, hippo, hip- 465 519 dial telephone, diamondback, diamond shape, diaper, digital clock, digital popotami, hippopotamus, hob, hockey puck, hockey stick, hog, hognose 466 520 watch, dingo, dining set, dining table, dinosaur, dior, dirt bike, dishrag, snake, calais, holden rodeo, holster, home theater, accord, 467 521 dishwasher, disk brake, doberman, dobro, doby, dock, doctor, doctor who, honda aerodeck, honda atv, honda cbr, honda city zx, honda civic, honda 468 522 dodge, dodge dakota, dodge durango, dodge fargo, dodge route , dodge cr85r, honda cross tour, honda cross tourer, honda fcx clarity, honda hsc, 469 523 sprinter van, dodge suburban, dog, dogsled, dome, dongxiao, donkervoort, honda incentives, honda mclaren, honda nsx, honda pilot, honda prelude, 470 524 donkey kong, donkey shrek, door handle, doormat, dori, dots, double honda rvf, honda smx, honda stream, honda vfr, honda vtr, honeycomb, 471 525 decker, dough, doumbek, dove, dowitcher, downbeat, dragon, dragonfly, hook, hoopskirt, horizontal bar, horizontal cylinder, horizontal lines, horn- 472 526 drake, drawer, drill, drilling platform, drum, drums percussion, drumstick, bill, horned , horse, horse cart, hotdog, hot pepper, hot pot, hottentot, 473 527 dry, dryer, dry van, dubai tower, duck, duetto, dugong, dumbbell, dump hourglass, house finch, howler monkey, hsv gts, huevos rancheros, hula 474 528 truck, dung beetle, dungeness crab, duryea motor wagon, dutch oven, hoop, , , hummingbird, hunting horn, hurdy gurdy, hyena, 475 529 dwyane wade, eagle, eagle jaguar, ear, earbud, earphone, earth, earthstar, hyundai, ibex, ibizan hound, ibm mainframe, ice bear, ice cream, ice 476 530 easel, echidna, ed sheeran, , eft, eggnog, eggplant, eggplant parmigiana, lolly, ice texture, idakka, igor andreev, imac, impala, indian , in- 477 531 egyptian cat, eiffel tower, elantra, electrical outlet, electric fan, electric dian elephant, indigo bunting, indoor, indri, inifiniti, international s120, 478 532 guitar, electric locomotive, electric ray, elephant, elevator, ender wiggin, ipad, ip camera, iphone, ipod, iridescent shark, irish bagpipe, irish setter, 479 533 english foxhound, english setter, english springer, entertainment center, irish terrier, irish water spaniel, irish wolfhound, iron, ironman, isaac new- 480 534 entlebucher, envelope, eraser, eskimo dog, espresso, espresso maker, estu- ton, isopod, isuzu, isuzu axiom, isuzu dmax, italian greyhound, jacamar, 481 535 ary cod, estuary perch, european fire salamander, european gallinule, ewer, jack crevalle, jacket, jackfruit, jackfruits, jack o’ lantern, jaguar, jaguar 482 536 excavator, expedition, explorer sport, eye, f16, f18, face, face powder, fal- car, jaguar daimler, japanese spaniel, jar, jasmine, jay, jean, jeans, jeep, 483 537 con, false albacore, fan, farming, farro, faucet, feather boa, feather texture, jeepster, jeep wrangler, jellyfish, jennair, jensen healey, jersey, jewelfish, 484 538 fence, fern, ferrari, ferrari 400i, ferris wheel, festiva, fiat, fiat abarth, fiat jigsaw puzzle, jinrikisha, joe, , joystick, jug, juggler, juice, junco, 485 539

5 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

540 594 jupiter, kakapo, kart, , katy perry, kayak, keeshond, kelpie, kenworth trich, otter, otterhound, oud, outdoor, oven, overpass, overskirt, ox, ox- 541 595 t2000, kerry blue terrier, kettle, kettle corn, key, keyboard, amanti, kia cart, oxen, ox plough, oxygen mask, oyster, oystercatcher, packard patri- 542 596 venga, kilka, killer whale, kilt, kimono, kindle, kindle fire, kinect, king cian, packet, pacman, paddle, paddlewheel, padlock, paintbrush, pajama, 543 597 crab, king , , kitchen blender, kitchen sponge, kite, kit palace, pan, , panchax, panpipe, pantaleon, panther, papad, pa- 544 598 fox, kiwi fruit, knee pad, knife, knifejaw, knot, koala, kobuz, kofta, ko- per, paper towel, papillon, parachute, parallel bars, parasol, parfait, park 545 599 heru, komodo dragon, komondor, kpanlogo, krill, kungfu, kuvasz, lab bench, parking meter, parmesan roll, parsley, parsley potato, partridge, 546 600 coat, labrador retriever, lacewing, lada 2107, laddle, laddoo, ladle, lady- passenger car, pastry, patas, patio, pay phone, pc, pea, peach, peacock, 547 601 bug, lakeland terrier, lakeside, lamborghini, lamp, lampshade, lancia, lan- pear, pedestal, pedestrian, pedicab, pekinese, pele, pelican, pembroke, pen, 548 602 cia flaminia, landing gear, land rover, langur, lantern, lantra, laptop, lawn pencil box, pencil sharpener, penguin, pepper, perfume, kelisa, 549 603 mower, leaf beetle, leafhopper, leaning tower, leatherback turtle, lebron , persian cat, person, pesto pasta, petri dish, peugeot bip- 550 604 james, lefse, leg, lemon, lens cap, lentil salad, leonard pope, leonberg, per, peugeot coupe, peugeot van, pharmacist, photocopier, photo frame, 551 605 leopard, leopard grouper, leopard shark, lesser panda, letter opener, lexus, piano, pick, , picket fence, pickup, pier, pig, pigeon, pigfish, 552 606 lexus gs450h, lexus gs hybrid, lhasa, library, lifeboat, lighter, limo, limou- piggy bank, pillar, pill bottle, pillow, pineapple, ping pong ball, pink, pin- 553 607 sine, limpkin, lincoln, lincoln car, lincoln cosmopolitan, , wheel, pipe, pirate, pistol, pitcher, pittsburgh steelers, , pizza roll, 554 608 liner, lion, lionel messi, lionfish, lipstick, little blue heron, lizard, llama, plaid, plain, plane, planer, planetarium, plastic bag, plate, plate rack, plat- 555 609 loaf, loafer, locomotive, loggerhead, logging truck, logitech g15, long- ter, platypus, play dough, playstation, plover, plow, plum, plunger, ply- 556 610 board, longfin mako, long horned beetle, longnose skate, looxcie, lorikeet, mouth, plymouth challenger, plymouth conquest, pocket bike, podium, po- 557 611 loris, lakers, lotion, lotus eclat, loudspeaker, loupe, lufthansa laroid camera, pole, polecat, police van, polo ball, pomegranate, pomera- 558 612 plane, lumbermill, lungfish, lycaenid, lychee, , m50, macaque, macaw, nian, poncho, aztek, pool ball, pool table, pop bottle, popcorn ball, 559 613 macbook air, macbook pro, mackerel tuna, madagascar cat, magenta, ma- porcupine, porsche boxster, porsche carrera, porsche cayenne, pot, potala 560 614 gentis, magnetic compass, , mailbag, mailbox, maillot, malamute, palace, potato, potato chips, potpie, potter wheel, power drill, power saw, 561 615 malibu, malinois, mallard, maltese dog, man, manefish, mango dressing, power tools, prairie chicken, prayer rug, praying mantid, pretzel, princeps, 562 616 manhole cover, manjur, manta ray, mantel, mantis, maraca, marat safin, printer, prison, proboscis monkey, projectile, projector, promontory, pro- 563 617 marimba, marimbula, mariner hybrid, mario, marjoram, marmoset, mar- tein bar, proton, proton arena, proton perdana, ptarmigan, puck, puffer, 564 618 mot, mars, mars rover, maserati gransport, maserati merak, maserati quat- pug, pumpkin, punching bag, purple, purse, pylon, pyramid, quail, quar- 565 619 troporte, , mask, matchstick, maypole, mazda, mazda kaza- ter window motor, quattroporte, queue, quill, quilt, r2d2, rabbit, racer, 566 620 mai, mazda mps, mazda , mazda senku, mazdaspeed, maze, measur- racket, radiator, radio, radio telescope, railing, rain barrel, rainy weather, 567 621 ing cup, meat loaf, medicine ball, medicine chest, meerkat, megalith, meg- rajma, ram, rapeseed, rasna, rayban, razor, recreational vehicle, rectan- 568 622 amouth, melon, menu, menudo, mercedes benz, mercedes cla, mercury, gular shape, recycle bin, red, red backed sandpiper, redbone, red bream, 569 623 mercury truck, mesclun, mexican hairless, mey, miami heat, michael jor- red breasted merganser, red devil, redfish, red fox, red piranha, redshank, 570 624 dan, michelin, michelin man, mickey mantel, mickey mouse, microphone, red wine, red wolf, reel, reflex camera, refrigerator, regal tang, remote 571 625 microwave, microwave oven, military uniform, milk, milk can, milk shark, control, renault grand modus, renault megane, renault megane cabriolet, 572 626 , miniature donkey, miniature pinscher, miniature poodle, miniature renault megane scenic, renault samsung, repinique, rer, restaurant, re- 573 627 schnauzer, minibus, mini cabrio, mini cooper, minigun, miniskirt, mini- volver, rhino, rhinoceros beetle, rhodesian ridgeback, rifle, ringlet, ring- 574 628 van, mink, mirage , mirror, missile, mitsubishi, mitsubishi bravo, neck snake, river lamprey, road roller, roadster, roaster, , robot, rock 575 629 , mitsubishi vii, mitsubishi viii, mitten, mixer, mixing beauty, rock crab, rockets, rockfish, rocking chair, rock python, roller 576 630 bowl, mixr, mobile home, mobius strip, model t, modem, modern, mola coaster, rolling pin, rolls royce, rolls royce cloud, ronaldinho, rondador, 577 631 mola, monarch, monastery, mongoose, monitor, monkey, monkfish, moon- rooster, rotisserie, rottweiler, round shape, rover, rover mgf, royal stew- 578 632 fish, moped, morgan, mortar, mortarboard, mosque, mosquito, mosquito art tartan, rubber duck, rubber eraser, ruddy turnstone, ruffed grouse, rug, 579 633 net, motocompo, motorboat, motor scooter, mountain bike, mountainous, , rugged scene, rule, ruler, running shoe, running water, rural, 580 634 mountain sucker, mountain tent, mouse, mousetrap, moussaka, mouth, rx 8, saab, saab 97x, safe, safety pin, sailboat, saint bernard, salad, saleen 581 635 moving van, mud turtle, muffin, mulan, mullet snapper, mushroom, mu- s7, salmon, salmon bites, saltshaker, saluki, samoyed, samsung galaxy, 582 636 sical clock, musket, muzzle, mysterious, nail, nannygai, napkin, narrow, sandal, sandbar, sandglass, santa claus, santro, sarangi, saron, sarong, sat- 583 637 natural, nautilus, navigator, navy destroyer, neck brace, necklace, needle- urn, ball, sax, saxophone, scabbard, scacciapensieri, scaglietti, 584 638 fish, nematode, neptune, newfoundland, news anchor, nexus 4, ngoni, night scale, scalpel, scandal, scanner, scarf, schipperke, school bus, schooner, 585 639 snake, nikola tesla, nikon d3100, nikon d600, nikon d90, nile tilapia, nil- scion, scissor, scone, scooter, scoreboard, scorpion, scotch terrier, scot- 586 640 gai, nintendo 64, nipple, altima, nissan auto, nissan clipper, nissan tish deerhound, screen, screw, screwdriver, scuba diver, sea anemone, sea 587 641 condor, nissan diesel, nissan gtir, , norfolk terrier, northern cucumber, seadoo, sea eel, seagull, seahorse, sea lion, sealyham terrier, 588 642 snakehead, norwegian elkhound, norwich terrier, nose, notebook, nova, seashell, seashore, sea , sea snake, sea spider, seat belt, seat exeo, sea 589 643 oat bran, oatmeal, obelisk, oboe, ocarina, octavina, odometer, office chair, urchin, sedan, sennheiser, sewing machine, shanny, shanz, shark, shed, 590 644 oil filter, okra, old english sheepdog, , onion, opel ascona, opel sheep, shelf, shetland sheepdog, shield, shih tzu, shime daiko, shingles, 591 645 astra, opel astra gtc, opel vauxhall signum, open area, opera house, opirus, shiny texture, ship, shirt, shoe, shoe shop, shofar, shoji, shopping bas- 592 646 optima, optimus prime, orange, orangutan, oreo, organ, oscilloscope, os- ket, shopping cart, shotgun, shoulder pad, shovel, shovelnose, shower 593 647

6 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

648 702 cap, shower curtain, shrubbery, shubunkin, shuttlecock, siamang, siamese vegetable tempura, velvet, vending machine, venus, vera zvonareva, ver- 649 703 cat, siberian husky, siberian tiger, , siena cathedral, sieve, sild, rophon, vertical, vertical cylinder, vertical lines, vestment, viaduct, vice 650 704 silky terrier, silver, silver seatrout, sink, skateboard, skeleton, ski, skil, ski tool, vicuna, vijay singh, villu, vine snake, vintage guitar, violet, violin, 651 705 mask, skin texture, skoda forman, sks, skunk, sky, sleeping bag, slide rule, viper, viper snake, vizsla, voca, volcano, volkswagen, volkswagen beetle, 652 706 sliding door, slimy sculpin, slipper, slot, sloth bear, slr, slug, sma c31, volkswagen polo, volleyball, volvo bus, vor indicator, vulture, waffle iron, 653 707 smallie, smart car, smoke, snail, snailfish, snake, snakehead, snorkel, snow wagon, walker hound, walking stick, wall, wallaby, wall clock, wallet, 654 708 leopard, snowmobile, snowplow, snowy weather, soap dispenser, soccer wardrobe, warm scenery, warplane, warthog, washbasin, washer, washing 655 709 ball, soccer net, sock, sofa, softball bat, soft coated wheaten terrier, so- machine, watch, water bottle, waterbuck, water buffalo, water jug, water 656 710 lar dish, soldering iron, solo oboe, sombrero, sonata, sony cybershot, sor- ouzel, water snake, water tower, wave, waverunner, weakfish, weasel, web 657 711 rel, soup, soup bowl, space bar, space heater, space rover, space shuttle, site, weevil, weimaraner, welsh springer spaniel, west highland white ter- 658 712 spadefish, spaghetti bolognaise, spaghetti squash, sparling, sparrow, spat- rier, whale, wheel, whippet, whiptail, whisk, whiskey jug, whistle, white, 659 713 ula, speckled, speedboat, sphinx, spider, spiderman, spider monkey, spi- whitefish, white stork, white wolf, whiting, wide, wig, wild boar, win- 660 714 der web, spigola, spindle, spiny lobster, sponge, spoon, spoonbill, sports dow, window screen, window shade, windshield, windsor tie, wine, wine 661 715 car, sportsperson, sportster, spotlight, spotted salamander, sprinkler, square bottle, wing, wire, wire haired fox terrier, wiry, wobbegong, wok, wolf, 662 716 shape, squash ball, squash casserole, squirrel, squirrel monkey, ss 90, wolf spider, wolpertinger, woman, wombat, wooden , wooden spoon, 663 717 ssangyong rodius stavic, staffordshire bullterrier, stage, stair, standard poo- wood rabbit, wool, worm fence, wreck, xindi, yamaha, yawl, yellow, yel- 664 718 dle, standard schnauzer, stapler, star, starbucks, starfish, statue, statue of low clorox bottle, yellowfin bream, yellow lady slipper, yellow tail, yellow 665 719 liberty, steam engine, steam locomotive, steam whistle, steel arch bridge, tail fish, yoda, yorkshire terrier, yurt, zebra, zebra shark, zia syed, zinedine 666 720 steel drum, steep, stephansdom, stereo, sterling rover, stethoscope, steve zidane, zopf, zucchini. 667 721 jobs, stick, still water, stingray, stinkhorn, stockpot, stole, stone, stone- Later we found around 1/4 of the 2000+ categories are 668 722 henge, stone wall, stool, stopwatch, stove, strainer, strawberry, streetcar, related to vehicles (aeroplane, bicycle, boat, bus, car, motor- 669 723 street sign, stretcher, stripe texture, stroganoff, strumstick, studio couch, cycle, train), with a lot of them being specific car models. 670 724 stupa, sturgeon, subaru 1000, subaru liberty, subaru outback, subaru trezia, Therefore, this list is heavily skewed to vehicles. 671 725 submarine, suburban, subway, subway train, , suit, sukiyaki, sulphur 672 To train the GoogleA net, the following list of scene cat- 726 butterfly, sulphur crested cockatoo, sun, sunbeam alpine, sun bear, sun- 673 egories are also included: 727 dial, sunflower, sunglass, sunglasses, sunny weather, sun perch, sunscreen, 674 abbey, access road, afghanistan, agra, air base, airfield, airlock, airplane 728 supercar, , supernova, supersonic jet, surgeonfish, suspension 675 cabin, airport, airport entrance, airport terminal, airport ticket counter, 729 bridge, sussex spaniel, suv, suzuki forenza, suzuki forenza wagon, suzuki 676 , alcove, alley, alliance bank , ameriquest field, amphitheater, 730 freewind, suzuki fronte, swab, swan, sweatshirt, swimming trunks, swing, 677 amsterdam, amusement , amusement park, anechoic chamber, angel 731 switch, sword, sxt neon, syclone, symmetrical, syringe, tabby, table, ta- 678 stadium, apartment building, apple store, apse, aquarium, aquatic theater, 732 ble lamp, tabor drum, tahoe, tailed frog, taj mahal, tam tam, tank, tape 679 aqueduct, arcade, arch, archaelogical excavation, archive, argentina, ar- 733 measure, tape player, taphon, tarabuka, tarantula, tart, tartan, tarte tatin, 680 lington stadium, armory, army base, arrival gate, art gallery, artificial lake, 734 tarwhine, tasmanian devil, tata nano, tattoo, tautog, taxi, teacup, teapot, 681 artists loft, art school, art studio, ashbourne, assembly line, , 735 teddy, teeth, telehandler, telephone, telescope, television, tempo vidal, 682 athens, athletic field, atlanta fulton county stadium, atrium home, atrium 736 tench, tennis ball, tennis racket, terrapin, tesla, tesla tower, tgv, thatch, the- 683 public, attic, att park, auditorium, auto factory, auto mechanics, auto racing 737 ater curtain, the bean, theremin, thimble, thornback ray, threadfish, three 684 paddock, auto showroom, autozone park, aylesbury, backstage, backstairs, 738 toed sloth, thresher, throne, thunder snake, tibetan mastiff, tibetan terrier, 685 badlands, badminton court, baggage claim, bakery kitchen, bakery shop, 739 tick, tiger, tiger beetle, tiger cat, , tiger woods, tile roof, timber 686 balcony, baldwin park, ballpark, ball pit, ballroom, bamboo forest, bank, 740 wolf, tinfoil barb, tire, titanic, titi, tmx, toaster, tobacco shop, tobu cycle, 687 bank vault, banquet hall, baptistry, bar, barbershop, barn, barndoor, barn- 741 toilet seat, toilet tissue, tomato, torch, , tortilla chip, tostadas, totem 688 yard, barrack, baseball diamond, baseball field, baseball stadium, base- 742 pole, toucan, tovshuur, towel, tower, tower bridge, tow truck, toy, 689 ment, basilica, basketball arena, basketball court, bathhouse, bathroom, 743 alphard hybrid, toyota auris hybrid, toyota belta yaris, , toy- 690 batters box, batting cage, battlement, bayou, bazaar, beach, beach house, 744 ota hiace, toyota , toyota mr2, toyota prius, toyota quantum, toy- 691 beauty salon, bedchamber, bedroom, beer garden, beer hall, beijing, belfry, 745 ota regius, toyota sienna, toyota sienna van, toyota spyder, toyota supra, 692 bell foundry, berlin, berth, berth deck, betting shop, bicycle racks, bindery, 746 toyota tarlet, toyota yaris, toy poodle, toyshop, toy terrier, tractor, traffic 693 biology laboratory, bistro, bleachers, bloomingdale, boardwalk, boat cargo 747 light, trailer, trailer truck, train, tram, tramontana, transformer, trash bin, 694 deck, boat deck, boathouse, bog, bolivia, bollywood, bologna, bomb shel- 748 tray, tree, tree frog, tree stump, trench coat, tribe, triceratops, tricycle, tri- 695 ter, bookbindery, bookstore, booth, botanical garden, bowling alley, bow 749 fle, trilobite, trimaran, tripod, triumphal arch, trolleybus, trombone, trout, 696 window, boxing ring, box seat, braves field, brazil, breakroom, brewery, 750 truck, truck rig, tub, tub gurnard, tumbler, tuna, tupan, turkey, turnstile, 697 brickyard, bridge, building complex, building facade, bullpen, bullring, 751 turquoise, turtle, tusker, tv, typewriter keyboard, uhu fish, uilleann pipe, 698 burial chamber, , bus depot, bus interior, bus shel- 752 ukulele, ultraman, umbrella, unicycle, upright, upright piano, uranus, ur- 699 ter, bus station, butchers shop, butte, cabana, cabin, cafeteria, call center, 753 ban, urn, utensil, vacuum, vacuum truck, valley, van, vase, vault, vauxhall 700 camden stadium, camden yards, campsite, campus, canal natural, canal 754 adam, vauxhall astra, vauxhall astravan, vauxhall corsa, vauxhall vectra, 701 urban, , candy store, canteen, canyon, caravansary, car 755

7 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

756 810 dealership, cardroom, cargo port, car interior, carport, carrousel, casino, chamber, levee, library, lido deck, lift bridge, lighthouse, limousine inte- 757 811 castle, catacomb, cathedral, catwalk, cavern, cemetery, cessna aircraft in- rior, liquor store, living room, loading dock, lobby, lock chamber, locker 758 812 terior, chalet, chaparral, chapel, chase field, checkout counter, cheese fac- room, loft, london, lookout station, los angeles coliseum, louisiana super- 759 813 tory, chemical plant, chemistry lab, chicken coop, chicken farm, childs dome, lumberyard, machine shop, maldives, manhole, mansion, manufac- 760 814 room, choir loft, church, circus tent, cisco field, citibank park, citizens tured home, market, marsh, martial arts gym, mastaba, maternity ward, 761 815 bank park, city, , classroom, clean room, cliff, cloak- mausoleum, mcafee coliseum, medina, memorial stadium, menhir, mesa, 762 816 room booth, cloakroom room, clock tower, cloister, closet, clothing store, mess hall, metrodome, , mezzanine, military hospi- 763 817 coast, coast road, cockpit, coffee shop, , comerica park, com- tal, military hut, military tent, mill, miller park, mine, mineshaft, mini 764 818 puter room, conference center, conference hall, conference room, confes- golf course, minute maid park, mission, moat dry, moat water, mobile 765 819 sional, construction site, control room, control tower, convenience store, home, monastery, moon bounce, moor, morgue, mosque, motel, moun- 766 820 coors field, corn field, corolla interior, corral, corridor, cottage, cottage gar- tain, mountain path, mountain road, movie theater, mudflat, museum, mu- 767 821 den, county stadium, courthouse, courtroom, courtyard, covered bridge, sic store, music studio, n3com park, nationals ballpark, nationals park, 768 822 cracker jack stadium, crawl space, creek, crevasse, crosley field, cross- natural history museum, naval base, network associates coliseum, news- 769 823 walk, cubicle library, cubicle office, cybercafe, dacha, dairy, dam, dance room, newsstand, new , new york, niagara falls, nightclub, 770 824 school, darkroom, day care center, delicatessen, dell diamond, dentists of- , nuclear power plant, nunnery, nursery, nursing home, oakland 771 825 fice, department store, departure lounge, desert road, desert sand, desert alameda county coliseum, , oasis, oast house, observa- 772 826 vegetation, diner, dinette home, dinette vehicle, dining car, dining hall, tory, ocean, office, office building, office cubicles, oilfield land, oilfield 773 827 dining room, dirt track, disch falk field, discotheque, distillery, ditch, dock, offshore, oil refinery, oilrig, old kent park, operating room, optician, or- 774 828 , dolmen, dolphins stadium, donjon, doorway, dorm room, chard, orchestra pit, organ loft, orlop deck, ossuary, outcropping, outhouse, 775 829 downtown, drainage ditch, dressing room, dress shop, drill rig, driveway, overpass, oyster bar, oyster farm, pac bell park, packaging plant, pagoda, 776 830 driving range, drugstore, dry dock, dubai, dugout, dunn tire park, dutchess palace, palace hall, palisade, pantry, paper mill, parade ground, paris, park, 777 831 stadium, earth fissure, ebbets field, edison field, editing room, egypt, elec- parking garage, parking lot, parlor, particle accelerator, party tent, pas- 778 832 trical room, electrical substation, elevated catwalk, elevator door, eleva- ture, patio, pavilion, pawnshop, pedestrian overpass, penalty box, perfor- 779 833 tor interior, elevator lobby, elevator shaft, embankment, embassy, engine mance arena, performance stadium, perfume shop, , pet shop, 780 834 room, entrance hall, equestrian, escalator, escarpment, estuary, excava- pharmacy, phoenix municipal stadium, phone booth, physics laboratory, 781 835 tion, exhibition hall, exposition park, fabric store, factory, fairway, farm, piano store, picnic area, pier, pig farm, pilothouse, pisa, pitchers mound, 782 836 fastfood restaurant, fence, , ferryboat, ferryboat cargo deck, pittsburgh, pizzeria, plane cargo deck, planetarium, plantation house, play- 783 837 ferryboat passenger deck, field cultivated, field road, field wild, fire es- ground, playroom, plaza, pnc park, podium, police station, , 784 838 cape, fire station, firing range, fish farm, fishmarket, fishpond, fitting room, pond, pontoon bridge, pool room, poop deck, porch, portico, portrait stu- 785 839 fjord, flea market, flight of stairs, floating dry dock, flood, florist shop, fly dio, postern, power plant, print shop, priory, progress energy park, progres- 786 840 bridge, food court, football arena, football field, football stadium, forbes sive field, promenade, promenade deck, pub, pulpit, pump room, putting 787 841 field, forest broadleaf, forest fire, forest needleleaf, forest path, forest road, green, quadrangle, qualcomm stadium, quay, quicksand, quonset hut, race- 788 842 formal garden, fort, fortress, foundry, fountain, freeway, freight elevator, course, raceway, raft, railroad track, railway yard, rainforest, ramp, ranch, 789 843 frontier field, funeral chapel, funeral home, furnace room, galley, game ranch house, reading room, reception, recreation room, rectory, recycling 790 844 room, gangplank, garage, garbage dump, gas station, gasworks, gate, gate- plant, refectory, repair shop, residential neighborhood, resort, rest area, 791 845 house, gazebo, store, geodesic dome, ghost town, gift shop, glacier, restaurant, restaurant kitchen, restaurant patio, restroom, revolving door, 792 846 glade, golf course, gorge, granary, great american ball park, great hall, rfk stadium, rice paddy, richmond county bank ballpark, riding arena, river, 793 847 greengrocery, greenhouse, grotto, guardhouse, gulch, gun deck, gun store, road cut, rock arch, rodeo arena, roller skating rink, rolling mill, rome, 794 848 gymnasium, hacienda, hallway, handball court, hangar, harbor, hardware roof, roof garden, root cellar, rope bridge, rosenblatt stadium, roundabout, 795 849 store, hatchery, hatchway, hat shop, hayfield, hayloft, hearth, heath, hedge roundhouse, rubble, ruin, runway, russ chandler stadium, sacristy, safeco 796 850 maze, hedgerow, heliport, herb garden, hhh metrodome, hi corbett field, field, plain, sandbar, sandbox, sand trap, , sauna, savanna, 797 851 highway, hill, , stadium, hockey arena, home of- sawmill, schoolhouse, schoolyard, science museum, , 798 852 fice, home theater, hoodoo, hospital, hospital room, hotel, hotel room, hot scriptorium, sea cliff, , seawall, security check point, server 799 853 spring, hot tub, house, hubert humphrey metrodome, hummingbird cen- room, sewer, sewing room, , shed, shipping room, shipyard, 800 854 tre, hunting lodge, hut, iceberg, ice cream parlor, ice floe, ice shelf, ice shoe shop, shopfront, shopping mall, shower, shower room, shrine, signal 801 855 skating rink, igloo, imaret, incinerator, industrial area, industrial park, inn, box, sinkhole, skate park, ski jump, ski lodge, ski resort, ski slope, sky, 802 856 interior savings centre, invesco field, irrigation ditch, islet, jacobs field, skydome, skyscraper, skywalk, slum, snowfield, snowy mountain, soc- 803 857 jacuzzi, jail, jail cell, japanese garden, jarry park, jetty, jewelry shop, cer arena, soccer field, soccer stadium, spa massage room, spa mineral 804 858 joss house, junk pile, junkyard, jury box, kansas city municipal stadium, bath, spillway, sporting goods store, squash court, stable, stadium, stage, 805 859 kart track, kasbah, kauffman stadium, kennel, keyspan park, kindergar- staircase, starting gate, steam plant, steel mill, stilt house water, stone cir- 806 860 den, kiosk, kitchen, kitchenette, korea, lab, labyrinth, lagoon, lake, lake cle, storage room, storeroom food, storm cellar, street, strip mall, strip 807 861 olmstead stadium, landfill, landing deck, las vegas, laundromat, lava flow, mine, student residence, submarine interior, subway interior, subway sta- 808 862 lavatory, lawn, , lean-to, lecture room, legends field, legislative tion, subway station entrance, subway station platform, sun deck, sun life 809 863

8 ICCV ICCV #465 #465 ICCV 2015 Submission #465. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

864 918 stadium, supermarket, , sushi bar, swamp, swimming hole, cat tabby, persian cat, angora, siamese cat, tiger cat, egyptian cat, lynx. 865 919 swimming pool, sydney, synagogue, taiga, target field, taxistand, taxiway, chair armchair, rocking chair, throne, bench, stool, folding chair, office 866 920 tea garden, tearoom, teashop, television room, television studio, tempe chair, aeron, barber chair, dentist chair. 867 921 diablo stadium, temple, tennis court, tent, terrace farm, textile mill, the- cow bull, oxen, ox, water buffalo. 868 922 ater, , thriftshop, throne room, tibet, ticket booth, ticket dining table table, desk. 869 923 window, , tobacco shop, tollbooth, toll plaza, topiary garden, 870 dog golden retriever, labrador retriever, german shepherd, eskimo dog, 924 tower, town house, toyshop, track, trading floor, trailer park, train depot, maltese dog, beagle, pug, pomeranian, samoyed, siberian husky, 871 925 train interior, train railway, train station, train station, train station platform, staffordshire bullterrier, english foxhound, malinois, chihuahua, 872 926 tree farm, tree house, trench, trestle bridge, tropicana field, tucson electric malamute, toy terrier, vizsla, shetland sheepdog, american stafford- 873 shire terrier, border collie, west highland white terrier, collie, boxer, 927 park, tundra, tunnel, tunnel car, tunnel rail, turkish bath, turner field, under- kelpie, dalmatian, rottweiler, saint bernard, papillon, flat coated 874 928 water coral reef, underwater ice, underwater kelp forest, underwater ocean, retriever, walker hound, brittany spaniel, greater swiss mountain 875 929 underwater pool, underwater wreck, us cellular arena, utility room, utility dog, bull mastiff, chow, miniature poodle, cocker spaniel, toy poo- 876 dle, rhodesian ridgeback, saluki, basset, greyhound, great pyrenees, 930 tunnel, valley, van interior, vegetable garden, velodrome, venice, venti- 877 welsh springer spaniel, lakeland terrier, basenji, miniature pinscher, 931 lation shaft, veranda, vestry, veterinarians office, viaduct, videostore, vil- japanese spaniel, kuvasz. 878 932 lage, vineyard, volcano, volleyball court, voting booth, waiting room, walk 879 horse NA 933 in freezer, warehouse, washhouse, watchtower, waterfall, waterfall frozen, 880 motorcycle motor scooter, yamaha, tmx, bsa, m50, dirt bike, honda vtr, 934 waterfall plunge, watering hole, water mill, water park, water tower, wa- honda cr85r, honda cross tourer, sportster, honda cbr. 881 935 ter treatment plant, wave, weighbridge, wendelstein, wet bar, wharf, wheat 882 person woman, actor, actress, man, girl, face, boy, bachelor, katy perry, 936 field, whispering gallery, widows walk, wig shop, wind farm, windmill, andre agassi, barack obama, andrew garfield, baby, caprice, ender 883 937 window seat, wine cellar, winery, winners circle, witness stand, woodland, wiggin, sportsperson. 884 938 workroom, workshop, wrestling ring, wrigley field, yankee stadium, yard, potted plant fern, parsley. 885 939 youth hostel, zen garden, ziggurat, zoo. sheep goat. 886 940 887 4.2. List for Category Expansion sofa couch, studio couch. 941 888 train bullet train, electric locomotive, locomotive, steam locomotive, 942 For category expansion (Section 3.3), we used more cat- steam engine, tgv, rer, subway train, passenger car, tram. 889 943 egories for each of the 20 PASCAL VOC classes. Since our 890 tv monitor tv, television, monitor, screen, imac. 944 list is skewed toward vehicles, some categories (e.g. bottle, 891 945 horse) did not receive any augmented category after seman- 892 References 946 tic verification. 893 [1] P. Agrawal, R. Girshick, and J. Malik. Analyzing the perfor- 947 894 aeroplane aircraft, fighter plane, cessna aircraft, plane, airplane, airliner, 948 supersonic jet, warplane, f16, boeing dreamliner, airbus 340, bi- mance of multilayer neural networks for object recognition. 895 plane, lufthansa plane, airbus a320, airbus a380, airbus a330, airbus In ECCV. 2014.2 949 896 330, delta airlines, american airlines. [2] X. Chen, A. Shrivastava, and A. Gupta. Neil: Extracting vi- 950 897 bicycle hero cycles, cannondale, bicycle built for two, bmx, mountain sual knowledge from web data. In ICCV, 2013.3 951 898 bike, giant defy. [3] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature 952 899 bird lorikeet, indigo bunting, goldfinch, bee eater, hummingbird, tou- hierarchies for accurate object detection and semantic seg- 953 900 can, broadbill, macaw, sparrow, chickadee, red backed sandpiper, mentation. In CVPR, 2014.1,2 954 901 magpie, baya weaver, pigeon, dove, house finch, brambling, eagle, [4] D. Hoiem, Y. Chodpathumwan, and Q. Dai. Diagnosing error 955 seagull, seagull, spoonbill, bald eagle, water ouzel, plover, jacamar, in object detectors. In ECCV. 2012.1,2 902 peacock, junco, hornbill, vulture, coucal, white stork, swan, african 956 903 grey, quail, partridge, bittern. [5] A. Quattoni and A. Torralba. Recognizing indoor scenes. In 957 CVPR, 2009.1 904 boat ship, container ship, schooner, sailboat, navy destroyer, catamaran, 958 905 aircraft carrier, titanic, speedboat, trimaran, motorboat, barge, fire- [6] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn 959 features off-the-shelf: an astounding baseline for recognition. 906 boat, lifeboat, gondola, canoe, yawl, kayak. 960 In CVPRW, 2014.1,2 907 bottle NA 961 [7] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, 908 bus school bus, double decker, articulated bus, coach bus, volvo bus, trol- 962 leybus, minibus, greyhound bus. S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, 909 et al. Imagenet large scale visual recognition challenge. 963 910 car , supercar, lamborghini, mazda, smart car, ferrari, hatch- arXiv:1409.0575, 2014.3 964 911 back, roadster, corvette, toyota prius, tesla, convertible, camaro, 965 porsche carrera, hyundai, saleen s7, buick, lincoln car, nissan auto, [8] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun 912 coupe, dodge, audi a8, fusion hybrid, sedan, viper, mitsubishi, ply- database: Large-scale scene recognition from abbey to zoo. In 966 913 mouth, scion, 1950 car, inifiniti, ac car, honda hsc, honda nsx, mazda CVPR, 2010.3 967 914 senku, rover, c max, nissan versa, acura, cabriolet, toyota supra, [9] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. 968 jaguar car, bmw 320, grigio telesto, chevrolet panther, citroen projet, 915 Learning deep features for scene recognition using places 969 fiat abarth, audi a6, vauxhall corsa, porsche boxster, toyota spyder, 916 plymouth challenger, gumpert apollo, suv, mazdaspeed, crossover, database. In NIPS, 2014.1,2 970 917 honda incentives, fiat, kia venga, citroen, lancia, datsun. 971

9