bioRxiv preprint doi: https://doi.org/10.1101/407452; this version posted September 4, 2018. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY 4.0 International license. Taxonomic Classification of Ants (Formicidae) from Images using Deep Learning Marijn J. A. Boer1 and Rutger A. Vos1;∗ 1 Endless Forms, Naturalis Biodiversity Center, Leiden, 2333 BA, Netherlands *[email protected] Abstract 1 The well-documented, species-rich, and diverse group of ants (Formicidae) are important 2 ecological bioindicators for species richness, ecosystem health, and biodiversity, but ant 3 species identification is complex and requires specific knowledge. In the past few years, 4 insect identification from images has seen increasing interest and success, with processing 5 speed improving and costs lowering. Here we propose deep learning (in the form of a 6 convolutional neural network (CNN)) to classify ants at species level using AntWeb 7 images. We used an Inception-ResNet-V2-based CNN to classify ant images, and three 8 shot types with 10,204 images for 97 species, in addition to a multi-view approach, for 9 training and testing the CNN while also testing a worker-only set and an AntWeb 10 protocol-deviant test set. Top 1 accuracy reached 62% - 81%, top 3 accuracy 80% - 92%, 11 and genus accuracy 79% - 95% on species classification for different shot type approaches. 12 The head shot type outperformed other shot type approaches. Genus accuracy was broadly 13 similar to top 3 accuracy. Removing reproductives from the test data improved accuracy 14 only slightly. Accuracy on AntWeb protocol-deviant data was very low. In addition, we 15 make recommendations for future work concerning image threshold, distribution, and 16 quality, multi-view approaches, metadata, and on protocols; potentially leading to higher 17 accuracy with less computational effort. bioRxiv preprint doi: https://doi.org/10.1101/407452; this version posted September 4, 2018. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY 4.0 International license. 2 BOER & VOS 18 Key words: Deep Learning, Convolutional Neural Network, Image Classification, 19 Formicidae, AntWeb, multi-view 20 21 The family of ants (Formicidae) is a large and diverse group within the insect order, 22 occasionally exceeding other insect groups in local diversity by far. Representing the bulk 23 of global biodiversity (Mora et al. 2011), ants are globally found (except on Antarctica) 24 and play important roles in a lot of ecosystems (H¨olldobleret al. 1990). As ants are found 25 to be good bioindicators, ecological and biodiversity data on them may be used to assess 26 the state of ecosystems (Andersen 1997; Andersen et al. 2002), which is important for 27 species conservation. Furthermore, insects are good surrogates for predicting species 28 richness patterns in vertebrates because of their significant biomass (Andersen 1997; 29 Moritz et al. 2001), even while using the morphospecies concept (Oliver et al. 1996; Pik 30 et al. 1999). To understand the ecological role and biological diversity of ants, it is 31 important to comprehend their morphology, and delimit and discriminate among species. 32 Even working with morphospecies, a species concept is still required for identification to 33 reach a level of precision sufficient to answer a research question. This is what is called 34 Taxonomic Sufficiency (Ellis 1985), which must be at a certain balance or level for a 35 research goal (Groc et al. 2010). Therefore, it is important to get a good understanding of 36 ant taxonomy, but many difficulties arise with the complicated identification of ants to 37 species level or to taxonomic sufficiency. 38 Ant taxonomy 39 Classifying and identifying ant species is complex work and requires specific 40 knowledge. While there is extensive work on this (e.g. Bolton (1994), Fisher et al. (2007), 41 and Fisher et al. (2016)), it is still in many instances reserved to specialists. To identify ant 42 species, taxonomists use distinct characters (e.g. antennae, hairs, carinae, thorax shape, 43 body shininess) that differ between subfamilies, genera, and species. However, the detailed bioRxiv preprint doi: https://doi.org/10.1101/407452; this version posted September 4, 2018. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY 4.0 International license. ANT IMAGE CLASSIFICATION USING DEEP LEARNING 3 44 knowledge on morphological characters can sometimes make species identification difficult. 45 Some ant species appear to be sibling species or very cryptic, and different castes 46 complicate things further. However, with a long history on myrmecological research, ants 47 are one of the best documented groups of insects and in recent years ant systematics have 48 seen substantial progress (Ward 2007). 49 Computer vision 50 In an effort to improve taxonomic identification, insect identification from images 51 has been a subject of computer vision research in the past few years. As some early papers 52 have shown (D. E. Guyer et al. 1986; Edwards et al. 1995; PJD Weeks et al. 1997; 53 PJ Weeks et al. 1999; Gaston et al. 2004), a promising start has been made on automated 54 insect identification, but there is still a long road to reaching human accuracy. Systems like 55 a Bayes classifier (D. E. Guyer et al. 1986) or DAISY ((PJ Weeks et al. 1999) mostly 56 utilized structures, morphometrics, and outlines. Together with conventional classifying 57 methods (such as a principal component analysis (PCA) (P Weeks et al. 1997)) images 58 data could be classified. Other, slightly more complex systems use simple forms of machine 59 learning (ML) (Kang et al. 2012), such as a support vector machine (SVM) ((Yang et al. 60 2015) or K-nearest neighbors (Watson et al. 2004). An identification system for insects at 61 the order level (including ants within the order of Hymenoptera) designed by Wang et al. 62 (2012b), used seven geometrical features (e.g. body width) and reached 97% accuracy. 63 Unfortunately, there are no classification studies that include ants, outside of the work of 64 Wang et al. (2012b) on insect order level, but for other insect groups, promising results 65 have been reported. Butterflies families (Lepidoptera) have been identified using shape, 66 color and texture features, exploiting the so-called CBIR algorithm (Wang et al. 2012a). 67 Insect identification to species level is harder, as some studies have shown. Javanese 68 butterflies (Lepidoptera: Nymphalidae, Pieridae, Papilionidae, and Riodinidae) could be 69 discriminated using the BGR-SURF algorithm with 77% accuracy (Vetter 2016). Honey bioRxiv preprint doi: https://doi.org/10.1101/407452; this version posted September 4, 2018. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY 4.0 International license. 4 BOER & VOS 70 bees (Hymenoptera: Apidae) could be classified with good results (>90%), using wing 71 morphometrics with multivariate statistics (Francoy et al. 2008). Gerard et al. (2015) could 72 discriminate haploid and diploid bumblebees (Hymenoptera: Apidae) based on differences 73 in wing shape (e.g. wing venation patterns) with great success (95%). Seven owlfly species 74 (Neuroptera: Ascalaphidae) were classified using an SVM on wing outlines (99%) (Yang 75 et al. 2015). Five wasp species (Hymenoptera: Ichneumonidae) could be classified using 76 PCA on wing venation data (94%) (P Weeks et al. 1997). Wen et al. (2012) classified eight 77 insect species (Tephritidae and Tortricidae) using 54 global morphological features with 78 86.6% accuracy. And Kang et al. (2012) fed wing morphometrics for seven butterfly species 79 (Lepidoptera: Nymphalidae and Papilionidae) in a simple neural network to classify, 80 resulting in >86% accuracy. However, a significant disadvantage in these systems is the 81 need for metric morphological features exploitation, which still require human expertise, 82 supervision, and input. 83 Deep learning 84 Deep learning (DL) may therefore be a promising taxonomic identification tool, as 85 it does not require human supervision. DL allows a machine to learn representations of 86 features by itself, instead of conventional methods where features need manual 87 introduction to the machine (Bengio et al. 2012; LeCun et al. 2015). In the past few years, 88 DL has attracted attention in research and its methods and algorithms have greatly 89 improved, which is why its success will likely grow in the future (LeCun et al. 2015). A 90 successful DL algorithm is the convolutional neural network (CNN), mostly used for image 91 classification and preferably trained using GPUs. These computationally-intensive 92 networks are designed to process (convolve) 2D data (images), using typical neural layers 93 as convolutional and pooling layers (Krizhevsky et al. 2012; LeCun et al. 2015) and can 94 even work with multi-view approaches (Zhao et al. 2017). A simple eight layer deep CNN 95 has strongly outperformed conventional algorithms that needed introduced features (Held bioRxiv preprint doi: https://doi.org/10.1101/407452; this version posted September 4, 2018. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY 4.0 International license.