Color-Based Segmentation of Sky/Cloud Images from Ground
Total Page:16
File Type:pdf, Size:1020Kb
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. XX, NO. XX, XX 2016 1 Color-based Segmentation of Sky/Cloud Images From Ground-based Cameras Soumyabrata Dev, Student Member, IEEE, Yee Hui Lee, Senior Member, IEEE, and Stefan Winkler, Senior Member, IEEE Abstract—Sky/cloud images captured by ground-based cam- Segmentation is one of the first steps in sky/cloud image eras (a.k.a. whole sky imagers) are increasingly used nowadays analysis. It remains a challenging task because of the non- because of their applications in a number of fields, including cli- rigid, feature-less, and poorly-defined structure of clouds, mate modeling, weather prediction, renewable energy generation, and satellite communications. Due to the wide variety of cloud whose shape also changes continuously over time. Thus, types and lighting conditions in such images, accurate and robust classical image segmentation approaches based on shape priors segmentation of clouds is challenging. In this paper, we present a [12] are not suitable. Furthermore, the wide range of lighting supervised segmentation framework for ground-based sky/cloud conditions (direct sunlight to completely covered skies) adds images based on a systematic analysis of different color spaces to the difficulty. and components, using partial least squares (PLS) regression. Unlike other state-of-the-art methods, our proposed approach is entirely learning-based and does not require any manually- A. Related Work defined parameters. In addition, we release the Singapore Whole Sky IMaging SEGmentation Database (SWIMSEG), a large As color is the most discriminating feature in sky/cloud database of annotated sky/cloud images, to the research com- images, most works in the literature use color for cloud munity. segmentation. Long et al. [13] showed that the ratio of red Index Terms—Cloud segmentation, whole sky imager, partial and blue channels from RGB color space is a good candidate least squares regression, SWIMSEG database for segmentation and tuned corresponding thresholds to create binary masks. Heinle et al. [14] exploited the difference of red and blue channels for successful detection and subsequent I. INTRODUCTION labeling of pixels. Liu et al. [15] also used the difference of red and blue channels in their superpixel-based cloud segmentation LOUDS have been extensively studied in the research framework. Souza et al. [16] used the Saturation (S) channel community over the past few decades. The analysis of C for calculating cloud coverage. Mantelli-Neto et al. [17] inves- clouds and their features is important for a wide variety of tigated the locus of cloud pixels in the RGB color model. Li et applications. For example, it has been used for nowcasting al. [18] proposed cloud detection using an adaptive threshold to deliver accurate weather forecasts [1], rainfall and satellite technique in the normalized blue/red channel. Yuan et al. precipitation estimates [2], in the study of contrails [3], and [19] proposed a cloud detection framework using superpixel various other day-to-day meteorological applications [4]. Yuan classification of image features. et al. have been investigating the clouds’ vertical structure [5] In these existing methods in the literature for cloud seg- and cloud attenuation for optimizing satellite links [6]. mentation, the selection of color models and channels has Sky/cloud imaging can be performed in different ways. not been studied systematically. Many existing approaches Satellite imagery [7], [8] and aerial photographs [9] are [14], [15], [18], [20], [21] use combinations of red and blue popular in particular for large-scale surveys; airborne light arXiv:1606.03669v1 [cs.CV] 12 Jun 2016 channels, which is a sensible choice, because the sky is detection and ranging (LiDAR) data are extensively used for predominantly blue due to the Rayleigh scattering of light aerial surveys [10]. However, these techniques rarely provide at shorter wavelengths. However, we are not aware of any sufficient temporal and/or spatial resolution for localized and experimental analysis presented regarding the efficacy of these short-term cloud analysis over a particular area. This is where color channels in sky/cloud image segmentation. Furthermore, ground-based whole sky imagers (WSIs) offer a compelling al- all of the above methods rely on manually-defined parameters ternative [11]. The images obtained from these devices provide and case-based decisions for segmentation. These make the high-resolution data about local cloud formation, movement, methods somewhat ad-hoc and prone to errors. Finally, most and other atmospheric phenomena. of them assign binary labels by design, which further reduces their flexibility and robustness. Manuscript received 11-Dec-2015; revised 17-Mar-2016; accepted 09-Apr- 2016. S. Dev and Y. H. Lee are with the School of Electrical and Elec- B. Outline of Our Contribution tronic Engineering, Nanyang Technological University, Singapore (e-mail: [email protected], [email protected]). The motivation of this paper is to propose a robust frame- S. Winkler is with the Advanced Digital Sciences Center (ADSC), work for color-based cloud segmentation under any illumi- University of Illinois at Urbana-Champaign, Singapore (e-mail: [email protected]). nation conditions, including a systematic analysis of color Send correspondence to S. Winkler, E-mail: [email protected]. channels. The framework is based on partial least squares IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. XX, NO. XX, XX 2016 2 (PLS) regression and provides a straightforward, parameter- In the existing literature, mainly color channels c1−9 [16], free supervised segmentation method. We show that our ap- [17] and c13−15 [13], [14], [18] have been used for sky/cloud proach is robust and offers a superior performance across image segmentation. In addition to these, we also consider ∗ ∗ ∗ two different databases as compared to current state-of-the- L a b space (c10−12) because of its perceptual uniformity art algorithms. Furthermore, it allows annotating each pixel properties as well as chroma (c16), because clouds tend to be with a degree of belongingness to the sky or cloud category, achromatic. instead of the usual binary labeling. In our previous work [22], we presented an analysis of color B. Principal Component Analysis channels for sky/cloud images captured by whole-sky cameras, which is an important pre-requisite for better segmentation. We use Principal Component Analysis (PCA) to determine The fuzzy c-means clustering method we used in that work the underlying structure of an image represented by the however suffers from similar shortcomings as other existing 16 color channels from Table I and analyze the inherent cloud segmentation methods. The main novel contributions of correlations amongst these components. Consider a sample the present manuscript compared to our earlier work include: image Xi of dimension m × n from a dataset consisting of N images (i = 1; 2; :::; N). The individual color channels • Introduction of a large public sky/cloud image database c1−16 are extracted for Xi and reshaped into column vectors with segmentation masks; mn×1 cj 2 IR where j = 1; 2; ::; 16. The cj obtained from the • Extensive evaluation of color components and selection e e different color channels of the sample image Xi are stacked of appropriate color channels on two different sky/cloud mn×16 alongside, to form the matrix X^ 2 IR . image databases; i ^ • Robust learning-based framework for sky/cloud segmen- Xi = [ec1; ec2; ::; ecj::; ec16]; where i = 1; ::; N (1) tation that outperforms existing methods. We normalize X^ with the mean c¯ and standard deviation The rest of this paper is organized as follows. Section II i j σ of the individual color channels of the image X , and introduces the color spaces under consideration and describes cj i represent it by X¨ : the statistical tools used for subsequent evaluation. Section i III discusses the supervised probabilistic segmentation frame- ce1 − c¯1 ce2 − c¯2 cej − c¯j cf16 − c¯16 X¨i = [ ; ; ::; ; ::; ] (2) work. The sky/cloud databases used for evaluation, including σc1 σc2 σcj σc16 our new SWIMSEG database, are presented in Section IV. Subsequently, we compute the covariance matrix M for An exhaustive analysis of color channels is performed in i each of X¨ . The covariance matrices M for each of the images Section V. Section VI presents the experimental evaluation i i provide interesting insights on how correlated the different of the segmentation framework, followed by a discussion of color channels are, and on the inherent relationships between the results in Section VII. Section VIII concludes the paper. the various color channels. The eigenvalue decomposition of Mi yields II. APPROACH &TOOLS st t st Mie = λ e ; (3) In this section, we describe the color models and channels we consider in this paper and present the statistical tools where est represents the sth eigenvector for tth eigenvalue, for evaluating their usefulness in sky/cloud image analysis. and λt represents the tth eigenvalue. These eigenvalues and Specifically, we use Principal Component Analysis (PCA) to eigenvectors are used for computing the relative contributions check the degree of correlation between the color channels of different color channels to the new orthogonal axes. and to identify those that capture the most variance. Loading As an indication of which color channel may be most factors from the primary principal component as well as the suitable for sky/cloud segmentation, we analyze the relative Receiver Operating Characteristic (ROC) curve for a simple loading factors of the different color channels on the primary thresholding applied directly to the color values of a given principal component. We calculate the loading factors by channel serve as indicators of a channel’s suitability for cloud considering the relative contribution of the different color classification. channels on the primary principal component.