A Visual Database of Recognisable Kitchen Utensils David Fullerton (s1137636) 4th Year Project Report Computer Science School of Informatics University of Edinburgh 2016 3 Abstract This report is the culmitaive summation of project ”A Visual Database of Recognis- able Kitchen Utensils”, from its early planning stages, to the more applicable stages in which issues and challenges were addressed, over come, or simply accepted, and finally the resulting work in which a classifier has been successfully created. Ulti- mately the goal of this project was to make the first steps towards a more efficient, or convenient home through the means of technology being applied to menial tasks, and whilst many issues have indeed been faced, and much progress can still, and always will be obtainable, ultimately that goal has been reached. To achieve this goal there were three main areas outlined to be tackled as a means of accomplishing a positive outcome: data collection, website creation and finally utensil recognition. Table of Contents 1 Introduction 7 1.1 Project Goals . 7 1.2 Summary of Contributions . 8 2 Dataset 9 2.1 Design . 9 2.2 Implementation . 10 2.2.1 Data Collection . 10 2.2.2 Website . 11 2.3 Discussion . 13 3 Thresholding and Description 15 3.1 Design . 15 3.2 Implementation . 16 3.2.1 Removing the Background . 16 3.2.2 Creating the Feature Vectors . 19 3.2.3 Testing Moment Invariance . 20 3.3 Discussion . 21 4 Recognition 23 4.1 Design . 23 4.2 Implementation . 24 4.3 Discussion . 26 5 Conclusion 29 5.0.1 Data collection . 29 5.0.2 Website . 29 5.0.3 Classifier, Utensil Recognition . 30 5.0.4 Final Thoughts . 30 6 Appendix 33 Bibliography 75 5 Chapter 1 Introduction In the future it is highly likely that we will have even more forms of convenience in our homes. This has been a process of many decades already, and is still happening at present. Should this trend of convenience within the home continue, as seems likely, robots being within the home does not sound outwith the bounds of possibility. One area in particular that seems prime for such an application is the kitchen; taking into account all the forms of convenience already implemented there anyway. The kitchen is in many ways, the heart of the home, and a room that everyone within said home uses. Therefore, if a robot was put within that room, it could simplify the area, as well as make it far more efficient. This could be done in a number of ways, the robot could perform such menial tasks such as washing up, drying, putting away cutlery and dishes. Or perhaps even preparing meals in their entirety. Such a robot with such responsibilities would be required to be able to recognise different kitchen utensils, and from there, know how to use them appropriately. In order to achieve a robot capable of such things, a database of images had to be collected, and a recognition algorithm created; this project has aimed to do both those things. By collecting such images and applying them to the algorithm, this project aims to make the first few steps towards the Kitchen of the future. The algorithm itself will aim to identify, in each image collected, what class that image of a utensil belongs to. Along with this, a simple website has been created to display the images collected. This allows the means to download them at convenience and use them within the project, and have them applied to an algorithm whenever needed. 1.1 Project Goals The goal for this project is ultimately to have created a database of images and a classi- fier that can both be used in the creating of a robot that can function within the kitchen and distinguish between different classes of utensils. • Firstly a database of kitchen utensil images had to be collected. Each image in turn had to be segmented in order to leave a binary image of just the utensil to 7 8 Chapter 1. Introduction be registered and understood by the program, and eventually robot. To create a broad database that would eventually allow for less margin of error, the in- tent was to collect approximately 20 different classes of utensil. Such utensils are as follows:bottle opener, dessert spoon, dinner fork, dinner knife, fish slice, kitchen knife ladle, masher, peeler, pizza cutter, potato peeler, serving spoon, soup spoon, spatula, tea spoon, tongs, whisk, wooden spoon, can opener and bread knife. Preferably each image and utensil would be unique, and 100 im- ages of each class would be taken. In doing so, a database of 2000 images would have been created for the algorithm to be applied to. • Secondly a website needed to be created. This website would provide links to download either individual images, or alternatively zip files containing either all raw images, all binary images or indeed all images in their entirety. These zip files were to be available for each class of utensils individually or all classes combined. Essentially it was to be designed in order to make access to the stock images easier. • Thirdly, and lastly, an algorithm was to be created. This visual classification algorithm would be intended to be trained to take a new image, from one of above listed classes, and recognise which class it belonged to. This allowing the recognition of a fork compared to a knife per say, or indeed a potato masher to a spatula. 1.2 Summary of Contributions • The collection of 449 images by manually photographing kitchen utensils. • Segmenting each image collected via image thresholding. • Creation of a simple xml based website that displays all images, both photo and binary segmented image, and provide means for downloading various sets of these images. These sets being either all photos of a class, all binary images of a class, all images of a class, all photos, all binary images or all images. • Adapted code from the course Introduction to Vision and Robotics, [2], as well as create code from scratch to create and store feature vector descriptions of every image in the database. Scatter plots and gaussian distributions were created to represent these feature vectors. • A multivariant Bayesian classifier was created which was trained by the feature vectors and would attempt to classify an image of a kitchen utensil presented to it. Chapter 2 Dataset 2.1 Design With such a large amount of data to be collected, as well as created, a certain amount of pre-planning had to be done in order to ensure the success of this project. The two main areas of planning were for area concerning the images; both in collecting them, and in putting them onto a website. For collecting the images two main avenues were considered; either by going out and photographing them all, and collecting them manually, or by acquiring them online. Both ideas raised challenges. The photography avenue was challenging in so far as the sheer mount of hours that would be have to be spent taking said photographs, ensuring their backgrounds were useable, as well as lighting and such too. Whereas the online avenue posed the challenge of copyright infringement and a tricky process of avoiding such wrong doing, along with this was vetting the images quality or lack thereof. Furthermore, once having collected said images, they had to be stored as well as made easily accessible. This was easiest done in the form of a website, and this too needed to be designed specifically for the projects needs. The images were to be made easily available for download from this website, and essentially this was all the website was required to do. This fact made it far less important for the website to be aesthetically pleasing, or cluttered by unnecessary functions; allowing a streamlined approach to its design. The focus here, was the images and access to them. It was decided that an xml file with a xslt to translate it into html format was appropriate as this allowed a more straightforward process of displaying all the images collected, which would amount to hundreds of images on the website. The overall layout of the website was inspire by Dermofit, [1], which had a similar basic function in that it also displayed a large array of images both segmented and not. The kitchen utensil was to have its own homepage, which is ”kitchen utensils.xml”, with a list of each class of utensil available along with an example image, as well as a segmented image of each class. A short description of the site, along with links to zip file download of all images, were also to be on this same homepage. From there a visitor of the site can easily navigate to any individual class page. On each of 9 10 Chapter 2. Dataset those class pages; all images of that class and their segmented counterparts are to be shown. As well as this, from the class pages it is possibly to download the following: individual images, a zip file of all raw photos, a zip filed of all segmented images, or a zip file of all images both segmented and raw. 2.2 Implementation 2.2.1 Data Collection This project involved the collection of a large amount of images containing kitchen utensils. Initially the plan was to gather the majority of the images required from various websites on the internet. By doing so, the time expended on manually taking photos individually would be taken to the minimum whilst still accomplishing a large amount of images gathered in a fraction of the time.