Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) Automatic Grassland Degradation Estimation Using Deep Learning Xiyu Yan1;∗ , Yong Jiang1;2 , Shuai Chen3;∗ , Zihao He1 , Chunmei Li4 , Shu-Tao Xia1;2 , Tao Dai1;2 , Shuo Dong4 and Feng Zheng5;† 1Dept. of Computer Science and Technology, Tsinghua University 2PCL Research Center of Networks and Communications, Peng Cheng Laboratory 3Baidu, Inc. 4Dept. of Computer Technology and Applications, Qinghai University 5Dept. of Computer Science and Engineering, Southern University of Science and Technology [email protected], fjiangy, [email protected], [email protected] Abstract is of top priority for protecting grassland ecosystem from de- sertification. Grassland degradation estimation is essential to The emergence of indicator plants is an important sign of prevent global land desertification and sandstorms. grassland degradation [Zhao et al., 2004]. Many countries Typically, the key to such estimation is to measure have successfully used specific plant species as indicators the coverage of indicator plants. However, tradi- for estimating grassland degradation [Mansour et al., 2016; tional methods of estimation rely heavily on hu- Mansour et al., 2012]. Our case study of degrading grassland man eyes and manual labor, thus inevitably lead- in Qinghai-Tibet Plateau demonstrates that as the grassland ing to subjective results and high labor costs. In degrades, the coverage of Stellera chamaejasme (SC) gradu- contrast, deep learning-based image segmentation ally accumulates. Thus, SC is regarded as the indicator plants algorithms are potentially capable of automatic for grassland degradation. Specifically, grassland would go assessment of the coverage of indicator plants. through five degradation stages before desertification, with Nevertheless, a suitable image dataset comprising the coverage of SC building up in each stage [Zhao et al., grassland images is not publicly available. To 2004], as shown in Table 1. Thus, it is intuitive to estimate this end, we build an original Automatic Grassland the grassland degradation stage based on the coverage of SC. Degradation Estimation Dataset (AGDE-Dataset), However, existing methods rely heavily on observations of with a large number of grassland images captured human eyes and manual labor, thus leading to subjective re- from the wild. Based on AGDE-Dataset, we are sults and high labor costs, which is undesirable in practice. able to propose a brand new scheme to automati- Consequently, there is an urgent need for developing an effec- cally estimate grassland degradation, which mainly tive and efficient method to automatically estimate the grass- consists of two components. 1) Semantic segmen- land degradation stage without human any interactions. tation: we design a deep neural network with an improved encoder-decoder structure to implement To do this, we attempt to leverage deep learning to cal- semantic segmentation of grassland images. In ad- culate automatically the coverage of SC in real-world grass- dition, we propose a novel Focal-Hinge loss to alle- land images based on a semantic segmentation algorithm, viate the class imbalance of semantics in the train- and then estimate the stage of grassland degradation by the ing stage. 2) Degradation estimation: we provide coverage of SC based on the results of recognition. Many the estimation of grassland degradation based on challenges stand in the way of achieving an automatic esti- the results of semantic segmentation. Experimen- mation of degradation. First, existing public datasets [Mot- tal results show that the proposed method achieves taghi et al., 2014; Cordts et al., 2016; Caesar et al., 2018; satisfactory accuracy in grassland degradation esti- Ros et al., 2016] contain substantially insufficient grassland mation. images and thus fail to provide us with enough samples to train the network. In addition, the aerial or satellite images used in the studies of remote sensing and environmental sci- 1 Introduction ences [Wang et al., 2018] are not high-resolution enough to capture such a tiny target as SC.Moreover, due to the particu- In recent years, massive grassland ecosystem has undergone larity of the grassland scene, capturing images with semantic degradation because of climatic variations and overgrazing, class imbalance is inevitable. Finally, existing semantic seg- thus resulting in multifarious ecological problems, such as mentation networks cannot handle directly the complex task desertification and sandstorms [Zhan et al., 2017]. Therefore, with these challenges. how to estimate the stage of grassland degradation accurately To this end, we first design a deep neural network to im- ∗Equal Contribution plement semantic segmentation that could accurately seg- †Contact Author ment the foreground (SC) from the background (grassland 6028 Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) Degradation Stage Coverage of SC the indicator plants in an image, we propose a semantic seg- mentation methodology. I 0%-19% II 20%-39% 2.2 Plant Density Estimation III 40%-59% IV 60%-79% In recent years, there has been a lot of research into plant den- V 80%-95% sity estimation based on image processing [Liu et al., 2017a; Liu et al., 2017b; Jin et al., 2017]. For example, [Liu et al., Table 1: The relationship between stage of grassland degradation 2017b] takes wheat plant images by a high-resolution RGB and coverage of Stellera chamaejasme (SC). camera and train Artificial Neural Networks with 10 manu- ally extracted features to estimate the number of plants. [Jin et al., 2017] captures images by a UAV and train a Sup- and other elements in a grassland scene) of the grassland im- port Vector Machine with 13 hand-crafted features to identify age at the pixel level. Aiming at the problem of sample in- wheat plant. sufficiency, we create a labeled dataset for automatic grass- In fact, plant density estimation entails the quantification land degradation estimation with ground-level grassland im- of the plant within a given unit area, which is highly biased ages captured from Qinghai-Tibet Plateau. To alleviate the by plant distribution. However, plant coverage refers to a rel- problem of class imbalance, we combine the advantages of ative area covered by the plant species in a plot, the calcu- reducing the class imbalance in Focal Loss [Lin et al., 2017] lation of which is more complex than that of quantification, and increasing class distance in smoothed Hinge loss [Rennie since the area would not necessarily scale with the quantity and Srebro, 2005], and propose an original Focal-Hinge loss of plant. In addition, both [Liu et al., 2017b] and [Jin et function. Next, we calculate the coverage of SC in the grass- al., 2017] treat the density estimation problem as object clas- land area through the analysis of the results from semantic sification, which might work with manually extracted fea- segmentation of grassland images and accordingly determine tures. In contrast, our task is based on semantic segmentation, the degradation stage according to the relationship between where hand-engineered features are not feasible, so we auto- stage and coverage (Table 1). matically extract features with the designed deep network. Through these two steps, we manage to automatically es- timate grassland degradation based on deep learning. To the 2.3 Semantic Segmentation best of our knowledge, we are the first to leverage deep learn- Semantic segmentation necessitates object classification at ing techniques to solve ecological problems regarding grass- the pixel level. Recently, there are many fabulous seman- land ecosystem. To be more specific, we propose a brand new tic segmentation models such as FCN [Long et al., 2015], scheme for grassland degradation estimation using semantic SegNet [Badrinarayanan et al., 2017], DeepLab-v3 [Chen segmentation by a deep neural network. Moreover, we design et al., 2018a], PSPNet [Zhao et al., 2017] and etc. Leong a Focal-Hinge loss function to train the proposed network for first proposed Fully Convolutional Networks (FCN) [Long et addressing the problem of class imbalance. Experiments of al., 2015] that is a convolutional network for dense predic- our scheme on the Automatic Grassland Degradation Estima- tion without a fully-connected layer. This model makes it tion Dataset (AGDE-Dataset) are carried out and reveal sat- possible to segment images at any size effectively, and it is isfying estimation results, which substantiate the feasibility much faster than traditional methods based on patch classifi- and prospect to solve the problem of automating grassland cation. However, an obtrusive problem using convolutional degradation estimation leveraging deep learning. neural networks for semantic segmentation is that pooling layers enlarge the receptive field, aggregating contextual in- 2 Related Work formation while discarding location information. Therefore, in order to solve this problem, an encoder-decoder architec- 2.1 Plant Identification ture is devised. The encoder gradually reduces the spatial Image-based plant identification is one of the most promis- dimensions by pooling layers, while the decoder restores the ing solutions towards furthering botanical taxonomy, as illus- target details and spatial dimensions step by step. Among trated by the wealth of research regarding this topic [Cerutti such architectures, U-Net [Ronneberger et al., 2015] is a very et al.,