Parsing Clothing in Fashion Photographs
Total Page:16
File Type:pdf, Size:1020Kb
Parsing Clothing in Fashion Photographs Kota Yamaguchi M. Hadi Kiapour Luis E. Ortiz Tamara L. Berg Stony Brook University Stony Brook, NY 11794, USA kyamagu, mkiapour, leortiz, tlberg @cs.stonybrook.edu { } Abstract In this paper we demonstrate an effective method for Shorts parsing clothing in fashion photographs, an extremely chal- lenging problem due to the large number of possible gar- ment items, variations in configuration, garment appear- ance, layering, and occlusion. In addition, we provide a large novel dataset and tools for labeling garment items, to Blazer enable future research on clothing estimation. Finally, we present intriguing initial results on using clothing estimates to improve pose identification, and demonstrate a prototype application for pose-independent visual garment retrieval. T-shirt T-shirt 1. Introduction Consider the upper east sider in her tea-dress and pearls, the banker in his tailored suit and wingtips, or the hipster in his flannel shirt, tight jeans, and black framed glasses. Our Figure 1: Prototype garment search application results. choice of clothing is tightly coupled with our socio-identity, Query photo (left column) retrieves similar clothing items indicating clues about our wealth, status, fashion sense, or (right columns) independent of pose and with high visual even social tribe. similarity. Vision algorithms to recognize clothing have a wide va- examining the problem in limited domains [2], or recogniz- riety of potential impacts, ranging from better social under- ing only a small number of garment types [4, 13]. Our ap- standing, to improved person identification [11], surveil- proach tackles clothing estimation at a much more general lance [26], computer graphics [14], or content-based im- scale for real-world pictures. We consider a large number age retrieval [25]. The e-commerce opportunities alone are (53) of different garment types (e.g. shoes, socks, belts, huge! With hundreds of billions of dollars being spent on rompers, vests, blazers, hats, ...), and explore techniques to clothing purchases every year, an effective application to accurately parse pictures of people wearing clothing into automatically identify and retrieve garments by visual sim- their constituent garment pieces. We also exploit the re- ilarity would have exceptional value (see our prototype gar- lationship between clothing and the underlying body pose ment retrieval results in Fig 1). In addition, there is a strong in two directions – to estimate clothing given estimates of contextual link between clothing items and body parts – pose, and to estimate pose given estimates of clothing. We for example, we wear hats on our heads, not on our feet. show exciting initial results on our proposed novel clothing For visual recognition problems such as person detection or parsing problem and also some promising results on how pose identification, knowing what clothing items a person is clothing might be used to improve pose identification. Fi- wearing and localizing those items could lead to improved nally, we demonstrate some results on a prototype visual algorithms for estimating body configuration. garment retrieval application (Fig 1). Despite the potential research and commercial gains of clothing estimation, relatively few researchers have ex- Our main contributions include: plored the clothing recognition problem, mostly focused on A novel dataset for studying clothing parsing, consist- • 1 null shorts shoes purse top necklace hair skin (a) Superpixels (b) Pose estimation (c) Predicted Clothing Parse (d) Pose re-estimation Figure 2: Clothing parsing pipeline: (a) Parsing the image into Superpixels [1], (b) Original pose estimation using state of the art flexible mixtures of parts model [27]. (c) Precise clothing parse output by our proposed clothing estimation model (note the accurate labeling of items as small as the wearer’s necklace, or as intricate as her open toed shoes). (d) Optional re- estimate of pose using clothing estimates (note the improvement in her left arm prediction, compared to the original incorrect estimate down along the side of her body). ing of 158,235 fashion photos with associated text an- tion from an underlying body contour, learned from training notations, and web-based tools for labeling. examples using principal component analysis to produce An effective model to recognize and precisely parse eigen-clothing. Most recently attempts have been made to • pictures of people into their constituent garments. consider clothing items such as t-shirt or jeans as seman- Initial experiments on how clothing prediction might tic attributes of a person, but only for a limited number of • improve state of the art models for pose estimation. garments [4]. Different from these past approaches, we con- A prototype visual garment retrieval application that sider the problem of estimating a complete and precise re- • can retrieve matches independent of pose. gion based labeling of a person’s outfit, for general images Of course, clothing estimation is a very challenging with a large number of potential garment types. problem. The number of garment types you might observe Clothing items have also been used as implicit cues of in a day on the catwalk of a New York city street is enor- identity in surveillance scenarios [26], to find people in an mous. Add variations in pose, garment appearance, lay- image collection of an event [11, 22, 25], to estimate occu- ering, and occlusion into the picture, and accurate cloth- pation [23], or for robot manipulation [16]. Our proposed ing parsing becomes formidable. Therefore, we consider approach could be useful in all of these scenarios. a somewhat restricted domain, fashion photos from Chic- Pose Estimation: Pose estimation is a popular and well topia.com. These highly motivated users – fashionistas – studied enterprise. Some previous approaches have con- upload individual snapshots (often full body) of their outfits sidered pose estimation as a labeling problem, assigning to the website and usually provide some information related most likely body parts to superpixels [18], or triangulated to the garments, style, or occasion for the outfit. This allows regions [20]. Current approaches often model the body as us to consider the clothing labeling problem in two scenar- a collection of small parts and model relationships among ios: 1) a constrained labeling problem where we take the them, using conditional random fields [19, 9, 15, 10], or dis- users’ noisy and perhaps incomplete tags as the list of pos- criminative models [8]. Recent work has extended patches sible garment labels for parsing, and 2) where we consider to more general poselet representations [5, 3], or incorpo- all garment types in our collection as candidate labels. rated mixtures of parts [27] to obtain state of the art results. 1.1. Related Work Our pose estimation subgoal builds on this last method [27], Clothing recognition: Though clothing items determine extending the approach to incorporate clothing estimations most of the surface appearance of the everyday human, in models for pose identification. there have been relatively few attempts at computational Image Parsing: Image parsing has been studied as a step recognition of clothing. Early clothing parsing attempts fo- toward general image understanding [21, 12, 24]. We con- cused on identifying layers of upper body clothes in very sider a similar problem (parsing) and take a related ap- limited situations [2]. Later work focused on grammati- proach (CRF based labeling), but focus on estimating la- cal representations of clothing using artists’ sketches [6]. belings for a particularly interesting type of object – people Freifeld and Black [13] represented clothing as a deforma- – and build models to estimate an intricate parse of a per- • HEATHER GRAY DOUBLE BREASTED VINTAGE SWEATER Fig 3). We make use of the tag portion of this meta-data to • CREAM FOREVER 21 SOCKS extract useful information about what clothing items might • BROWN SEYCHELLES BOOTS • CREAM SELF MADE POSTLAPSARIA DRESS be present in each photo (but can also ignore this infor- • BROWN BELT • BLACK COACH BAG mation if we want to study clothing parsing with no prior • ROMANTIC // KNEE HIGHS // VINTAGE // knowledge of items). Sometimes the tags are noisy or in- BOOTS // DRESS // CREAM // COMFY // CARDIGAN // SELF MADE // AWESOME complete, but often they cover the items in an outfit well. BOOTS // DOLL // WARM // ADORABLE // COLD WEATHER // CUUUTEE As a training and evaluation set, we select 685 photos with good visibility of the full body and covering a variety • HOLIDAY SWEATER STYLE FOR BRUNCH IN FALL 2010 of clothing items. For this carefully selected subset, we de- sign and make use of 2 Amazon Mechanical Turk jobs to Figure 3: Example Chictopia post, including a few photos gather annotations. The first Turk job gathers ground truth and associated meta-data about garment items and styling. pose annotations for the usual 14 body parts [27]. The sec- If desired, we can make use of clothing tags (underlined) as ond Turk job gathers ground truth clothing labels on super- potential clothing labels. pixel regions. All annotations are verified and corrected if necessary to obtain high quality annotations. In this ground truth data set, we observe 53 different son’s outfit into constituent garments. We also incorporate clothing items, of which 43 items have at least 50 image discriminatively trained models into the parsing process. regions. Adding additional labels for hair, skin, and null 1.2. Overview of the Approach (background), gives a total of 56 different possible cloth- We consider two related problems: 1) Predicting a cloth- ing labels – a much larger number than considered in any ing parse given estimates for pose, and 2) Predicting pose previous approach [4, 2, 6, 26, 11, 22, 25]. On average, given estimates for clothing. Clothing parsing is formu- photos include 291.5 regions and 8.1 different clothing la- lated as a labeling problem, where images are segmented bels. Many common garment items have a large number into superpixels and then clothing labels for every segment of occurrences in the data set (number of regions with each are predicted in a CRF model.