Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset
Total Page:16
File Type:pdf, Size:1020Kb
Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset Menglin Jia?1, Mengyun Shi?1;4, Mikhail Sirotenko?3, Yin Cui?3, Claire Cardie1, Bharath Hariharan1, Hartwig Adam3, Serge Belongie1;2 1Cornell University 2Cornell Tech 3Google Research 4Hearst Magazines Abstract. In this work we explore the task of instance segmentation with attribute localization, which unifies instance segmentation (detect and segment each object instance) and fine-grained visual attribute cat- egorization (recognize one or multiple attributes). The proposed task requires both localizing an object and describing its properties. To illus- trate the various aspects of this task, we focus on the domain of fashion and introduce Fashionpedia as a step toward mapping out the visual aspects of the fashion world. Fashionpedia consists of two parts: (1) an ontology built by fashion experts containing 27 main apparel categories, 19 apparel parts, 294 fine-grained attributes and their relationships; (2) a dataset with everyday and celebrity event fashion images annotated with segmentation masks and their associated per-mask fine-grained at- tributes, built upon the Fashionpedia ontology. In order to solve this challenging task, we propose a novel Attribute-Mask R-CNN model to jointly perform instance segmentation and localized attribute recogni- tion, and provide a novel evaluation metric for the task. Fashionpedia is available at: https://fashionpedia.github.io/home/. Keywords: Dataset, Ontology, Instance Segmentation, Fine-Grained, Attribute, Fashion 1 Introduction Recent progress in the field of computer vision has advanced machines' ability to recognize and understand our visual world, showing significant impacts in fields including autonomous driving [52], product recognition [32,14], etc. These real-world applications are fueled by various visual understanding tasks with the goals of naming, describing (attribute recognition), or localizing objects within an image. Naming and localizing objects is formulated as an object detection task (Fig- ure1(a-c)). As a hallmark for computer recognition, this task is to identify and indicate the boundaries of objects in the form of bounding boxes or segmentation masks [12,37,17]. Attribute recognition [6,29,9,36] (Figure1(d)) instead focuses ? equal contribution. 2 M. Jia et al. <"/0)A =>&?)@$0)@0+5% ,).1$0 !,"+. 2+.1,)@>#)"/$)A Apparel Categories Lounge Puffer (Shorts) (Jacket) / Down (Jacket) Cargo B#&55)A%/0&3,A)# (Shorts) Harem Sweatpants (Pants) Argyle / Jogger (Pants) Henley Fine-grained Attributes !,"+. (Shirt) Culottes Tube (Top) / Safari Apparel Categories C)13,"#%D'+$E Washed Varsity bandeau (Jacket) / (Jacket) Plant (Top) Utility Sheath (Jacket) / Animal (Dress) Cargo Boardshorts C)13,"#%D'+$E Buckle / Board 26;;)$#+9", Umbrella (Shorts) Trunks Fine-grained Attributes =>&?)@$0)@0+5% Giraffe Hobble Tunic (Top) (Skirt) / Culotte Wiggle ,).1$0 Epaulette(Shorts) (Skirt) HAS_ATTRIBUTE Blouson Umbrella Buckle (Dress) Track HAS_ATTRIBUTE (Jacket) Snakeskin TRIBUTE TTRIBUTE A (pattern) HAS_AT Tank (Top) HAS_ Peacock Pea (Jacket) Tuxedo Gypsy Smock (Top)(Jacket) (Skirt) / Broomstick Epaulette HAS_ATTRIBUTE (Skirt) Classic CardiganMandarin (with Gown (Collar) / Crop (Top) / military Nehru Rivet Midriff (Top) !,"+. (Jacket) opening)(Collar) Hip-huggers (Pants) / Hip Biker 26;;)$#+9", Norfolk huggers (… Jodhpur / (Jacket) / (Jacket) Breeches Moto (Pants) Bomber (Jacket) =.:,)%,).1$0 (Jacket) Leggings 8&#;",%<"+/$ Cardigan (with Peg (Pants) Mao (Jacket) HAS_ATTRIBUTE Classic -,6%D45).+.1E Rugby (Shirt) (T-shirt) Rivet / Stud / Polo Banded Rivet Spike opening) (Collar) HAS_ATTRIBUTE (Collar) / Flat HAS_ATTRIBUTE HAS_ATTRIBUTE B+/$#)//)A% Knit (Collar) HAS_ATTRIBUTE HAS_ATTRIBUTE Houndstooth 2,+;%D'+$E% HAS_ATTRIBUTE (pattern) Anorak Rubber Hoodie Straw HAS_ATTRIBUTE HAS_ATTRIBUTE Feather Skater (Skirt) Stripe Metal HAS_ATTRIBUTE Watch Fair Isle Kimono Wrap Sheath (Skirt) HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTEHAS_ATTRIBUTE Camisole (Dress) Crop (Pants) Floral Shearling Frayed Robe Trench (Coat) Nehru HAS_ATTRIBUTE Trucker (Coat) (Jacket) UTE HAS_ATTRIBUTE (Jacket) / Printed RIB HAS_ATTRIBUTE HAS_ATTRIBUTE Denim HAS_ATTRIBUTE Stand-away Tutu (Skirt) / (Collar) (Jacket) Ballerina HAS_ATTRIBUTE Windbreaker HAS_ATT (Skirt) Sailor Wood Ringer Fringe Chanel HAS_ATTRIBUTE Watch HAS_ATTRIBUTE (T-shirt) HAS_ATTRIBUTE (Jacket) Duster HAS_ATTRIBUTEHAS_ATTRIBUTE (Pants) HAS_ATTRIBUTE Kaftan / HAS_ATTRIBUTE (Coat) HAS_ATTRIBUTE Plain HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Bloom Caftan HAS_ATTRIBUTE HAS_ATTRIBUTE (pattern) Berm HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Dropped-sh… -ers E !"#$ !%# !&#$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!'# HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE -uda HAS_ATTRIBUTE (Shorts) HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE High HAS_ATTRIBUTE HAS_ATTRIBUTE RIBUT Glove HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE BUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Peg Halter HAS_ATTRIBUTE HAS_ATTRIBUTE Smocking / _ATT HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Low Shirring Cheetah Napoleon ATTRI HAS_ATTRIBUTE Fringe HAS_ATTRIBUTE HAS_ATTRIBUTE _ (Lapel) HAS_ATTRIBUTE HAS_ATTRIBUTE HAS (Top) Bead Flame HAS_ATTRIBUTE AS HAS_ATTRIBUTE HAS_ATTRIBUTE H HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE -nco HAS_ATTRIBUTE (Skirt) HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Ruched / HAS_ATTRIBUTE no special Chained HAS_ATTRIBUTE Ruching manufactur… Military HAS_ATTRIBUTE 3.*%450% Swing (Coat) (Opening) HAS_ATTRI HAS_ATTRIBUTE Flare HAS_ATTRIBUTE HAS_ATTRIBUTE Distressed / HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Glove Cargo HAS_ATTRIBUTE Embossed Ripped (Coat) E Dot BUTE HAS_ATTRIBUTE (Skirt) Three HAS_ATTRIBUTE IBUT HAS_ATTRIBUTE TTR A HAS_ATTRIBUTE E HAS_ATTRIBUTE S_ HAS_ATTRIBUTE quarter HAS_ATTRIBUTE HA Capri (Pants) HAS_ATTRIBUTE HAS_ATTRIBUTE IBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE (length) / HAS_ATTRIBUTE HAS_ATTRIBUTE IBUT HAS_ATTRIBUTE HAS_ATTRIBUTE Peasant HAS_ATTRIBUTE R TTR HAS_ATTRIBUTE Duffle UTE HAS_ATTRIBUTE (Top) HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRI HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE (Coat) / HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE E HAS_A HAS_ATTRIBUTE E Duffel (Co… HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE ATTRIB HAS_ATTRIBUTE HAS_ATT TRIBUT IBUT Polo (Shirt) Tie-up Shift HAS_ATTRIBUTE HAS_ATTRIBUTE Shearling HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE TR (Shorts) Bootcut / HAS_ATTRIBUTE S_AT Puffer (Coat) HAS_ A BUTE HAS_ATTRIBUTE AT Boot cut TE H Basque HAS_ATTRIBUTE HAS_ATTRIBUTEHAS_ATTRIBUTE E HAS_ATTRIBUTE HAS_ATTRIBUTE (Dress) HAS_ATTRIBUTE HAS_ATTRIBUTE AS_ BU I (wasitline) HAS_ATTRIBUTE H HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE ATTRIBUTE Bow _ HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE AS HAS_ATTRIBUTE TE H HAS_ATTRIBUTE DeepFashion2HAS_ATTRIBUTE HAS_ATTRIBUTE ModaNet _ATTR HAS_ATTRIBUTE BUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE _ATTRIBUT HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE IBU HAS_ATTRIBUTE HAS_ATTRIBUTE Tea (Dress) HAS_ATTRIBUTE Lapel HAS HAS ATTRI HAS_ATTRIBUTE UTE HAS_ATTRIBUTE HAS_ATTRIBUTE B HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE AS_ HAS_ATTRIBUTE HAS_ATTRIBUTE E HAS_ATTRIBUTE H Chevron TTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE UTE HAS_ATTRIBUTE AS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE U-neck / U HAS_ATTRIBUTE HAS_ATTRIBUTE ATTRIBUTE HAS_ATTRIBUTE H HAS_ATTRIBUTE HAS_ATTR HAS_A HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Cheongsams HAS_ATTRIBUTE Kilt HAS_ATTRIBUTE (Neck) HAS_ATTRIBUTE HAS_ HAS_ATTRI HAS_ATTRIBUTE HAS_ATTRIBUTE E HAS_ATTRIBUTE HAS_A HAS_ATTRIBUTE H HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE _ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE T S / Qipao Single AS_AT HAS_ATTRIBUTE U Tight (fit) / HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE UTEHA HAS_ATTRIBUTEHAS_ATTRIBUTEHAS_ATTRIBUTE IB T breasted HAS_ATT HAS_ATTRIBUTE S_ATTRIB HAS_ATTRIBUT IBUTE E TRIB Slim (fit) / R HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE T TE HAS_ATTRIBUTE HA TR HAS_ATTRIBUTE IBUTE HAS_ATTRIBUTE Skinny (fit) HAS_ATTRIBUTE E HAS_ATTRIBUTE IBU UT RIBUTE HAS_ATTRIBUTE HAS_ATTRIB HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE UT HAS_ATTRIBUTEE HAS_ATTRIBUTE Under HAS_ATTRIBUTETE HAS_ATTRIBUTE E HAS_ATTR HAS_ATTRIBUTE Raglan HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTR HAS_ATTRIBUTEHAS_ATTRIBUTE Bow HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATT HAS_ATTRIBUTE UT HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTEHAS_ATTRIBUTEHAS_ATTRIBUTE (T-shirt) HAS_ATTRIBUTE TRIB HAS_ATTRIBU Oversized HAS_ATTRIBUTE DeepFashion2 TE HAS_ATTRIBU HAS_ATTRIBUTE HAS_ATTRIBUTE ModaNetU HAS_ATTRIBUTE E Plastic HAS_ATTRIBUTEHAS_ATTRIBUTE HAS_ATTRIBUTE -shirt (Lapel) HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE UT HAS_ATTRIBUTE _AT HAS_ATTRIBUTE HAS_ATTRIBUTEHAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE Rah-rah Lapel Zip-up HAS_ATTRIBUTEHAS_ATTRIBUTE ATTRIB HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTEHAS_ATTRIBUTE HAS (Skirt) HAS_ATTRIBUTE HAS_ATTRIBUTE TE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTEHAS_ATTRIBHAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_HAS_ATTRIB HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTE HAS_ATTRIBUTEHAS_ATTRIBUTE