Identifying the Cuisine of a Plate of Food
Total Page：16
File Type：pdf, Size：1020Kb
Identifying the Cuisine of a Plate of Food Mabel Mengzi Zhang University of California San Diego [email protected] Abstract a network of common ingredients in the dishes. Ingredients are the core components that make a dish 1.2. Approach what it is, besides the preparation process. Using the idea An intuitive way that humans use to identify geographi- of attribute-based classification, we seek to classify plates cal cuisines, or even individual dishes, is by the types of in- of food to the correct cuisine by the country, using the in- gredients that form the dish and how they appear in amount, gredients as attributes for a plate of food. Because of the shape, and arrangement. We focus on the type of ingredi- important role that ingredients have in any plate of food, ents and their amount on a plate to approach the classifica- this method can be generalized to any type of dishes with tion, as they are the most fundamental components that any any type of ingredients, and can learn new dishes not seen entry-level cook would have to control in order to make a in the training, as long as the ingredients can be speci- dish what it is. Thus, our approach first tries to identify the fied to fit an attribute descriptor. Our dataset came from ingredients on the plate, then using the ingredients found, online sources and includes three cuisines, each with two it tries to classify which cuisine category the plate belongs dishes represented by 76 images. Even though our dataset to, according to the type of the ingredients and the amount is limited, reasonable results and a mean accuracy of 82.9% present. show that the method could be generalized to more cate- This ingredient-cuisine approach is a two-layer pro- gories. cess, and we take advantage of this structure by using the attribute-based classifier [8]. We represent the ingredients as attributes in the first layer, and the cuisine as the cate- 1. Introduction gories in the second layer. We will describe the approach in further details in the subsequent sections. 1.1. Context In section2, we discuss some of the existing works re- Food is an indispensible part of our lives. In today’s lated to food recognition or classification. In sections3 globalized market, food from different geographical regions and4, we explain the method and the implementation and show remarkable variations in the choice of ingredients and dataset we used. Then we present and analyze the results the ways to prepare them. Some cuisines have ingredi- in section5. Finally, possibilities for future work are sug- ent choices unimaginable to customers habituated to other gested in section6. cuisines, but still present surprisingly tasty dishes. Besides unique ingredients, even the same ingredients, depending 2. Related Work on the preparation process, may end up preserving different fractions of the nutritional value and very different calories. As far as we know, there are no previous work aiming to If restaurant networks provide a mobile app that can aim identify the cuisine on a plate of food. However, there are at a photograph of a plate of food and report its cuisine, some relevant food classification problems tha have been the composing ingredients, and list other similar dishes, it explored and had similar technical challenges. could be a good marketing strategy. In addition, restaurants 2.1. Identify ingredients on the dish of a specific cuisine can also refer to other cuisines’ use of ingredients to sparkle ideas for more creative dishes that Color and texture are the most obvious properties that utilize similar ingredients in new ways. human use to identify food. Bolle et al. have successfully One core component of such an application is to recog- classified produce using color, texture, and other features nize the cuisine and ingredients on a plate of food. Then the [3]. A recently published food recognition work by Yang et information may later be used to compare cuisines and find al. also identifies each ingredient on the plate before deter- 1 mining the category of the food, such as sandwich or salad 3. Method [12]. Therefore we also approach the problem by first iden- tifying the ingredients on the dish before classifying the cui- 3.1. High Level Image Information sine that the plate belongs to. We used color to cluster a plate of food into sections of different ingredients, and we used the amount of each type 2.2. Feature descriptor for the cuisine of ingredients on a plate to help determine the cuisine. Color is one of the most important pieces of information Yang et al.’s method first identified the ingredients, gave that humans use to identify ingredients. For example, while each ingredient a probability label, and then used pairwise whole zuchini and Chinese eggplant look very similar, we local features among the ingredients to determine the food know that the green one is the zuchini, and the purple one is category, by calculating the distance, orientation, and other the eggplant. It is even more true for food on a plate, where properties between each pair of ingredients [12]. The fea- the food is cut up into pieces, often similar small cubes or tures were used across 61 categories of food, where 7 cate- round slices, making it even more confusing to identify, if gories were analyzed in more detail [12]. color information is not available. Therefore we preserve Identifying cuisines is different from identifying the spe- the color information from the image and use methods that cific food, because different cuisines may have different can deal with color. Features such as Scale-Invariant Fea- typical arrangements on the plate that is recognizable with- ture Transform (SIFT) and SIFT-like descriptors discard out necessarily knowing all the ingredients composing the color information, and since we are currently only using one dish. For example, one does not have to know the com- type of low-level image feature, we did not look at features posites of a piece of sushi to be able to tell that a dish is that would discard color [9]. However, shape desciptors like Japanese. Some dishes have unique ingredients, such as SIFT are potentially helpful if the method is expanded to red ginger, that are indicative of the cuisine; however, some somehow include more than one low-level features, such as other dishes use commonly-seen ingredients, and it is the a combination of SIFT and RGB histograms. Details on particular preparation process that brings forth a character- how we used color to cluster a plate of food into ingredients istic appearance. For example, rice may be compressed into is discussed in section 4.1. tight round and triangular shapes in Japanese cuisine, but The amount of each ingredient present on a plate is an- eaten loose in most Indian and Chinese dishes, dyed in yel- other important factor. It makes the difference between a low in Indian dishes, and appear orange in Mexican dishes. plate of beef-stuffed large green peppers and a plate of bul- Given that each cuisine can have a variety of plate lay- gogi, or Korean stir-fried beef, with peppers on the garnish, outs, we will assume a typical appearance known to the as the amount of green pepper is more than or less than the mass. Since not very much literature focuses on food recog- amount of beef, respectively. We measure the amount of an nition, there are possibly unexplored feature descriptors that ingredient by taking the ratio of that ingredient’s area to the can yield promising results specifically to cuisines. entire plate’s food area, where the area is simply the number Wu has approached a similar problem with classifying of pixels. During the training process, an ingredient’s area dishes by ingredients, and the results presented suggest that ratio is used to train the attribute classifier for that ingredi- color textons yields a better result than color histograms and ent. More details follow. texture combined. [11]. Thus we took advantage of the idea and directly used color textons as clustering. The nature of 3.2. Attribute-Based Classification our problem is different from Wu’s in that we classify the As mentioned earlier, the attribute-based classifier is a type of cuisines from ingredients, whereas Wu classifies for two-layer method that uses high-level features called “at- ingredients. We also found a clustering program that yields tributes” as an intermediate layer between the raw image more suitable results for us than the JSEG used in Wu’s and the categorization results [8]. An attribute is a piece of experiments. factual information that can be described by high-level nat- Differing from all the work listed above, we use a two- ural language and can generalize to all the categories, such level approach, where instead of classifying the cuisine di- as “green” and “lives in water.” In our case, we use each rectly from the image, we introduce an intermediate high- ingredient as an attribute. After a certain number of specific level description of ingredients, which are used instead of attributes are chosen, an attribute vector is assigned to each the raw image features to identify the cuisine. The advan- category. For example, in a hypothetical two-attribute case tage of our high-level description is that it is more intuitive of “green” and “lives in water”, the binary attribute vector and can learn new categories very quickly without having to for salmon would be [0, 1], for zuchini would be [1, 0], and rerun the training, as long as the user can specify the high- for seaweed would be [1, 1].