Machine Vision and Applications manuscript No. (will be inserted by the editor) Natalia Larios · Hongli Deng · Wei Zhang · Matt Sarpola · Jenny Yuen · Robert Paasch · Andrew Moldenke · David A. Lytle · Salvador Ruiz Correa · Eric N. Mortensen · Linda G. Shapiro · Thomas G. Dietterich Automated Insect Identification through Concatenated Histograms of Local Appearance Features Feature Vector Generation and Region Detection for Deformable Objects Received: date / Accepted: date Abstract This paper describes a computer vision ap- aged via an apparatus that manipulates the specimens proach to automated rapid-throughput taxonomic iden- into the field of view of a microscope so that images are tification of stonefly larvae. The long-term goal of this obtained under highly repeatable conditions. The images research is to develop a cost-effective method for environ- are then classified through a process that involves (a) mental monitoring based on automated identification of identification of regions of interest, (b) representation of indicator species. Recognition of stonefly larvae is chal- those regions as SIFT vectors [25], (c) classification of the lenging because they are highly articulated, they exhibit SIFT vectors into learned “features” to form a histogram a high degree of intraspecies variation in size and color, of detected features, and (d) classification of the feature and some species are difficult to distinguish visually, de- histogram via state-of-the-art ensemble classification al- spite prominent dorsal patterning. The stoneflies are im- gorithms. The steps (a) to (c) compose the concatenated feature histogram (CFH) method. We apply three region detectors for part (a) above, including a newly developed Natalia Larios principal curvature-based region (PCBR) detector. This University of Washington, Department of Electrical Engi- neering detector finds stable regions of high curvature via a wa- E-mail: [email protected] tershed segmentation algorithm. We compute a separate dictionary of learned features for each region detector, H. Deng · W. Zhang · E. Mortensen · T. G. Dietterich Oregon State University, School of Electrical Engineering and and then concatenate the histograms prior to the final Computer Science classification step. E-mail: {deng, zhangwe, enm, tgd}@eecs.oregonstate.edu We evaluate this classification methodology on a task Matt Sarpola · Robert Paasch of discriminating among four stonefly taxa, two of which, Oregon State University, Department of Mechanical Engi- Calineuria and Doroneuria, are difficult even for experts neering to discriminate. The results show that the combination E-mail: [email protected] of all three detectors gives four-class accuracy of 82% Andrew Moldenke and three-class accuracy (pooling Calineuria and Doro- Oregon State University, Department of Botany and Plant neuria) of 95%. Each region detector makes a valuable Pathology E-mail: [email protected] contribution. In particular, our new PCBR detector is able to discriminate Calineuria and Doroneuria much David A. Lytle Oregon State University, Department of Zoology better than the other detectors. E-mail: [email protected] Keywords classification · object recognition · interest Jenny Yuen operators · region detectors · SIFT descriptor Massachusetts Institute of Technology, Computer Science and AI Laboratory E-mail: [email protected] 1 Introduction Salvador Ruiz Correa Children’s National Medical Center, Department of Diagnos- tic Imaging and Radiology There are many environmental science applications that E-mail: [email protected] could benefit from inexpensive computer vision meth- Linda G. Shapiro ods for automated population counting of insects and University of Washington, Department of Computer Science other small arthropods. At present, only a handful of and Engineering projects can justify the expense of having expert ento- E-mail: [email protected] mologists manually classify field-collected specimens to 2 Natalia Larios et al. obtain measurements of arthropod populations. The goal in a considerable loss of information and, potentially, in of our research is to develop general-purpose computer the failure to detect changes in water quality. vision methods, and associated mechanical hardware, for Besides its practical importance, the automated iden- rapid-throughput image capture, classification, and sort- tification of stoneflies raises many fundamental computer ing of small arthropod specimens. If such methods can vision challenges. Stonefly larvae are highly-articulated be made sufficiently accurate and inexpensive, they could objects with many sub-parts (legs, antennae, tails, wing have a positive impact on environmental monitoring and pads, etc.) and many degrees of freedom. Some taxa ex- ecological science [14,18,8]. hibit interesting patterns on their dorsal sides, but others are not patterned. Some taxa are distinctive, others are The focus of our initial effort is the automated recog- very difficult to identify. Finally, as the larvae repeat- nition of stonefly (Plecoptera) larvae for the biomoni- edly molt, their size and color change. Immediately after toring of freshwater stream health. Stream quality mea- molting, they are light colored, and then they gradually surement could be significantly advanced if an economi- darken. This variation in size, color, and pose means that cally practical method were available for monitoring in- simple computer vision methods that rely on placing all sect populations in stream substrates. Population counts objects in a standard pose cannot be applied here. In- of stonefly larvae and other aquatic insects inhabiting stead, we need methods that can handle significant vari- stream substrates are known to be a sensitive and robust ation in pose, size, and coloration. indicator of stream health and water quality [17]. Be- To address these challenges, we have adopted the bag- cause these animals live in the stream, they integrate wa- of-features approach [15,9,32]. This approach extracts a ter quality over time. Hence, they provide a more reliable bag of region-based “features” from the image without measure of stream health than single-time-point chem- regard to their relative spatial arrangement. These fea- ical measurements. Aquatic insects are especially use- tures are then summarized as a feature vector and classi- ful as biomonitors because (a) they are found in nearly fied via state-of-the-art machine learning methods. The all running-water habitats, (b) their large species diver- primary advantage of this approach is that it is invariant sity offers a wide range of responses to water quality to changes in pose and scale as long as the features can change, (c) the taxonomy of most groups is well known be reliably detected. Furthermore, with an appropriate and identification keys are available, (d) responses of choice of classifier, not all features need to be detected many species to different types of pollution have been in order to achieve high classification accuracy. Hence, established, and (e) data analysis methods for aquatic even if some features are occluded or fail to be detected, insect communities are available [6]. Because of these ad- the method can still succeed. An additional advantage is vantages, biomonitoring using aquatic insects has been that only weak supervision (at the level of entire images) employed by federal, state, local, tribal, and private re- is necessary during training. source managers to track changes in river and stream health and to establish baseline criteria for water qual- A potential drawback of this approach is that it ig- ity standards. Collection of aquatic insect samples for nores some parts of the image, and hence loses some biomonitoring is inexpensive and requires relatively little potentially useful information. In addition, it does not technical training. However, the sorting and identifica- capture the spatial relationships among the detected re- tion of insect specimens can be extremely time consum- gions. We believe that this loss of spatial information ing and requires substantial technical expertise. Thus, is unimportant in this application, because all stoneflies aquatic insect identification is a major technical bottle- share the same body plan and, hence, the spatial layout neck for large-scale implementation of biomonitoring. of the detected features provides very little discrimina- tive information. Larval stoneflies are especially important for biomon- The bag-of-features approach involves five phases: (a) itoring because they are sensitive to reductions in wa- region detection, (b) region description, (c) region clas- ter quality caused by thermal pollution, eutrophication, sification into features, (d) combination of detected fea- sedimentation, and chemical pollution. On a scale of or- tures into a feature vector, and (e) final classification ganic pollution tolerance from 0 to 10, with 10 being of the feature vector. For region detection, we employ the most tolerant, most stonefly taxa have a value of 0, three different interest operators: (a) the Hessian-affine 1, or 2 [17]. Because of their low tolerance to pollution, detector [29], (b) the Kadir entropy detector [21], and change in stonefly abundance or taxonomic composition (c) a new detector that we have developed called the is often the first indication of water quality degradation. principal curvature-based region detector (PCBR). The Most biomonitoring programs identify stoneflies to the combination of these three detectors gives better perfor- taxonomic resolution