Deep Learning-based Method for Classifying and Localizing Potato Blemishes Sofia Marino, Pierre Beauseroy and André Smolarz Institut Charles Delaunay/M2S, FRE 2019, Université de Technologie de Troyes, Keywords: Deep Learning, Potato Blemishes, Classification, Localization, Autoencoder, SVM. Abstract: In this paper we address the problem of potato blemish classification and localization. A large database with multiple varieties was created containing 6 classes, i.e., healthy, damaged, greening, black dot, common scab and black scurf. A Convolutional Neural Network was trained to classify face potato images and was also used as a filter to select faces where more analysis was required. Then, a combination of autoencoder and SVMs was applied on the selected images to detect damaged and greening defects in a patch-wise manner. The localization results were used to classify the potato according to the severity of the blemish. A final global evaluation of the potato was done where four face images per potato were considered to characterize the entire tuber. Experimental results show a face-wise average precision of 95% and average recall of 93%. For damaged and greening patch-wise localization, we achieve a False Positive Rate of 4.2% and 5.5% and a False Negative Rate of 14.2% and 28.1% respectively. Concerning the final potato-wise classification, we achieved in a test dataset an average precision of 92% and average recall of 91%. 1 INTRODUCTION is the difficulty to design a feature extractor adap- ted to each pattern, that require human expertise to Potato is one of the most important food crops con- suitable transform the raw input image into a good sumed all over the world with a total production that representation, exploitable to achieve the classifica- exceeds 374.000.000 tons (IPC, 2018). The physi- tion task. In the last few years, deep learning techni- cal aspect of this edible tuber is of great importance ques have demonstrated outstanding results in many in determining the market price between the diffe- research fields, such as image classification (Mohanty rent stages of the supply chain. Their quality is af- et al., 2016; Oppenheim and Shani, 2017; Picon et al., fected by different types of blemishes that may be 2018), object detection (Redmon et al., 2016), speech visually identified. In most cases, the quality cont- recognition (Hinton et al., 2012) and semantic seg- rol is still done manually by human operators, where mentation (Badrinarayanan et al., 2015). The main the main drawbacks are: subjectivity and high labor advantage of deep learning methods is their ability to costs. Thus, several inspection methods have been use raw data and automatically find the representa- developed to automate these tasks in a more efficient tion needed to achieve the classification or detection and cost-effective way. Computer vision and machine task. Deep Learning applied in agriculture is gro- learning techniques have been applied successfully wing rapidly with promising results (Mohanty et al., in the quality control of agricultural produce (Barnes 2016; Brahimi et al., 2017; Oppenheim and Shani, et al., 2010; Jhuria et al., 2013; Zaborowicz et al., 2017). Unfortunately, these methods mainly use a 2017). The first works were focused on computer pixel-labeled dataset which construction is laborious vision systems consisted of three main stages: fir- and time-consuming. For the remaining methods, stly, pre-processing of images acquired by cameras either they do not do blemish localization, i.e. they were done. Secondly, hand-crafted features were ex- do only classification, or an approximate localization tracted in order to obtain relevant information about using a patch scale too large. Furthermore, deep le- the object and finally, machine learning techniques arning based methods are not widely explored in po- were used to classify according to features extracted tato blemish detection. The main contributions of this (Miller and Delwiche, 1989; Bolle et al., 1996; Tao work are as follows: et al., 1995). The key problem with these systems 107 Marino, S., Beauseroy, P. and Smolarz, A. Deep Learning-based Method for Classifying and Localizing Potato Blemishes. DOI: 10.5220/0007350101070117 In Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods (ICPRAM 2019), pages 107-117 ISBN: 978-989-758-351-3 Copyright c 2019 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved ICPRAM 2019 - 8th International Conference on Pattern Recognition Applications and Methods • A large image-level labeled dataset that contains by shape, they compared the detected potato boun- 6 different classes was created with the help of dary with an ellipse which represented a good potato two experts including multiple varieties of pota- shape. The projected area of the potato and the minor toes and images taken using multiple camera de- axis of the fitted ellipse were also used for classifying vices. by weight and size respectively. The overall success rate was 86.5%. In (Noordam et al., 2000), a method • The created dataset was used to train an efficient for grading potatoes by size, shape and various de- Convolutional Neural Network, that classifies po- fects was introduced. Color and shape features were tato faces and also selects the images that require used to train a Linear Discriminant Analysis combi- further analysis. ned with Mahalanobis distance classifier. Eccentri- • A combination of autoencoder and SVMs is pro- city and central moments were used to differentiate posed to localize the damaged and greening areas between defects and diseases. Then, Fourier Descrip- in these selected images. tors were used to detect misshapen potatoes. Unfor- tunately, pixel-level labeled datasets were needed to • We introduced a global evaluation of the tuber ac- train and validate the models for each potato culti- cording to the previous results. var. Potato classification in good, rotten and green The paper is organized as follows: Section 2 presents was presented in (Dacal-Nieto et al., 2009). Features a brief related work. We detail our proposed method for every RGB and HSV channel were extracted using in Section 3. Discussion and results are presented in histograms and co-occurrence matrices. Then, feature Section 4. Finally, we conclude the paper in Section selection was applied using a genetic algorithm to fi- 5. nally classify potatoes with a nearest neighbor algo- rithm. Detection rate of 83.3%, 88.5% and 84.7% was achieved for good, rotten and green potatoes respecti- vely. (Barnes et al., 2010) introduced an AdaBoost 2 RELATED WORK based system to discriminate between blemished and non-blemish pixels. Color and texture features were Machine vision systems have been widely applied extracted and the best features for the classification to classification and blemish detection in agricultu- task were automatically selected by the AdaBoost al- ral produce. In (Bolle et al., 1996) authors propo- gorithm. They achieved a success rate of 89.6% for sed a system to classify fruits and vegetables in gro- white potatoes and 89.5% for red potatoes. Authors cery and supermarkets stores. Color and texture his- in (ElMasry et al., 2012) developed a real-time system tograms were used as input to a nearest-neighbour to detect irregular potatoes. Geometrical features and classifier achieving over 95% top-4 choices correct Fourier Descriptors were used as input to a Linear responses. A vectorial normalization was proposed Discriminant Analysis to identify the most relevant in (Vízhányó and Felföldi, 2000) to differentiate bet- features that were useful to characterize regular pota- ween the natural browning and the browning caused toes. A success rate of 98.8% for regular potatoes and by disease in mushrooms. In (Xing et al., 2005), a 75% for misshapen potatoes was achieved in a test method that use principal component analysis on hy- experiment. A method based on Principal Compo- perspectral images for determining apples as sound nent Analysis combined with one-vs-one SVM multi- or bruised was presented. An accuracy of about 93% classifier was proposed in (Xiong et al., 2017). They for detecting sound apples and 86% for bruised ap- attained an overall recognition rate of 96.6% for clas- ples was achieved. In (Blasco et al., 2007), the aut- sifying potatoes in normal, green, germinated and da- hors introduced a region-oriented segmentation algo- maged. Recently, various methods based on deep le- rithm to identify defects of citrus fruits. An accuracy arning have been applied to image analysis of agricul- of 95% was obtained. A main drawback of this met- ture produce. Authors in (Mohanty et al., 2016) used hod is that authors assumed that most surface of the a public PlantVillage dataset to identify 14 crop spe- fruit was of sound peel, which is not always the case. cies and 26 diseases. They analyzed the performance A banana segmentation method was proposed in (Hu of two CNN architectures: AlexNet and GoogLeNet. et al., 2014). The segmentation of the banana from The best accuracy achieved was 99.35% with the pre- the background and the detection of damaged lesi- trained GoogLeNet using a color dataset. Another ons were made by two k-means clustering algorithms. work on leaf disease classification was presented by Machine vision systems applied to potatoes were stu- (Brahimi et al., 2017). They fine-tuned a pre-trained died in several works. Authors in (Zhou et al., 1998) CNN to classify 9 different diseases in tomato lea- applied color thresholding in HSV color space to de- ves. They demonstrated that fine-tuning a pre-trained tect greening potatoes. In order to classify potatoes 108 Deep Learning-based Method for Classifying and Localizing Potato Blemishes CNN outperformed shallow models with hand-crafted features. In (Oppenheim and Shani, 2017) the authors proposed a method to classify patches of potatoes in five distinct classes: healthy, black dot, black scurf, silver scurf and common scab.