Exploiting weather forecast data for cloud detection

Abstract

Accurate, fast detection of clouds in satellite imagery has many applications, for example Numerical Weather Prediction (NWP) and climate studies of both the atmosphere and of the Earth’s surface temperature. Most operational techniques for cloud detection rely on the differences between observations of cloud and of clear-sky being more or less constant in space and in time. In reality, this is not the case - different clouds have different spectral properties, and different cloud types are more or less likely in different places and at different times, depending on atmospheric conditions and on the Earth’s surface properties. Observations of clear sky also vary in space and time, depending on atmospheric and surface conditions, and on the presence or absence of aerosol particles. The Bayesian approach adopted in this project allows pixel-specific physical information (for example from NWP) to be used to predict pixel-specific observations of clear sky. A physically-based, spatially- and temporally-specific probability that each pixel contains a cloud observation is then calculated. An advantage of this approach is that identification of ambiguously classed pixels from a probabilistic result is straightforward, in contrast to the binary result generally produced by operational techniques. This project has developed and validated the Bayesian approach to cloud detection, and has extended the range of applications for which it is suitable, achieving skills scores that match or exceed those achieved by operational methods in every case. High temperature gradients can make observations of clear sky around ocean fronts, particularly at thermal wavelengths, appear similar to cloud observations. To address this potential source of ambiguous cloud detection results, a region of imagery acquired by the AATSR sensor which was noted to contain some ocean fronts, was selected. Pixels in the region were clustered according to their spectral properties with the aim of separating pixels that correspond to different thermal regimes of the ocean. The mean spectral properties of pixels in each cluster were then processed using the Bayesian cloud detection technique and the resulting posterior probability of clear then assigned to individual pixels. Several clustering methods were investigated, and the most appropriate, which allowed pixels to be associated with multiple clusters, with a normalized vector of ‘membership strengths’, was used to conduct a case study. The distribution of final calculated probabilities of clear became markedly more bimodal when clustering was included, indicating fewer ambiguous classifications, but at the cost of some single pixel clouds being missed. While further investigations could provide a solution to this, the computational expense of the clustering method made this impractical to include in the work of this project. This new Bayesian approach to cloud detection has been successfully developed by this project to a point where it has been released under public license. Initially designed as a tool to aid retrieval of sea surface temperature from night-time imagery, this project has extended the Bayesian technique to be suitable for imagery acquired over land as well as sea, and for day-time as well as for night-time imagery. This was achieved using the land surface emissivity and surface reflectance parameter products available from the MODIS sensor. This project added a visible Radiative Transfer Model (RTM), developed at University of Edinburgh, and a kernel-based surface reflectance model, adapted here from that used by the MODIS sensor, to the cloud detection algorithm. In addition, the cloud detection algorithm was adapted to be more flexible, making its implementation for data from the SEVIRI sensor straightforward. A database of ‘difficult’ cloud and clear targets, in which a wide range of both spatial and temporal locations was represented, was provided by M´et´eo-France and used in this work to validate the extensions made to the cloud detection scheme and to compare the skill of the Bayesian approach with that of operational approaches. For night land and sea imagery, the Bayesian technique, with the improvements and extensions developed by this project, achieved skills scores 10% and 13% higher than M´et´eo-France respectively. For daytime sea imagery, the skills scores were within 1% of each other for both approaches, while for land imagery the Bayesian method achieved a 2% higher skills score. The main strength of the Bayesian technique is the physical basis of the differentiation between clear and cloud observations. Using NWP information to predict pixel-specific observations for clear-sky is relatively straightforward, but making such predictions for cloud observations is more complicated. The technique therefore relies on an empirical distribution rather than a pixel-specific prediction for cloud observations. To try and address this, this project developed a means of predicting cloudy observations through the fast forward-modelling of pixel-specific NWP information. All cloud fields in the pixel-specific NWP data were set to 0, and clouds were added to the profile at discrete intervals through the atmosphere, with cloud water- and ice- path (cwp, cip) also set to values spaced exponentially at discrete intervals up to saturation, and with cloud pixel fraction set to 25%, 50%, 75% and 100%. Only single-level, single-phase clouds were modelled, with the justification that the resulting distribution of predicted observations, once smoothed through considerations of uncertainties, is likely to include observations that would correspond to multi-phase and multi-level clouds. A fast RTM was run on the profile information for each of these individual clouds and cloud altitude-, cloud pixel fraction- and channel-specific relationships between cwp (and similarly cip) and predicted observations were calculated from the results of the RTM. These relationships were used to infer predicted observations for clouds with cwp/cip values other than those explicitly forward modelled. The parameters used to define the relationships were interpolated to define relationships for predicted observations of cloud at 10m vertical intervals through the atmosphere, with pixel coverage ranging from 25% to 100% in increments of 1%. A distribution of predicted cloud observations is then achieved without explicit forward-modelling of an impractical number of atmospheric states. Weights are applied to the representation of individual clouds within the final Probability Density Function (PDF) in order to make the distribution of predicted observations realistic, according to the pixel-specific NWP data, and to distributions seen in a global reference dataset of NWP profiles from the European Centre for Medium Range Weather Forecasting (ECMWF). The distribution is then convolved with uncertainties in forward-modelling, in the NWP data, and with sensor noise to create the final PDF in observation space, from which the conditional probability that the pixel observation corresponds to a cloud observation can be read. Although the relatively fast computational implementation of the technique was achieved, the results are disappointingly poor for the SEVIRI-acquired dataset, provided by M´et´eo-France, against which validation was carried out. This is thought to be explained by both the uncertainties in the NWP data, and the forward-modelling dependence on those uncertainties, being poorly understood, and treated too optimistically in the algorithm. Including more errors in the convolution introduces the problem of quantifying those errors (a non-trivial task), and would increase the processing time, making implementation impractical. In addition, if the uncertianties considered are too high then a PDF flatter than the empirical distribution currently used would be produced, making the technique less useful. It is hoped that advances in NWP will result in the implementation of this technique in the Bayesian cloud detection algorithm yielding improved results in the future. At present no clear improvement is seen and the computational expense of including the local cloud PDF calcluation in the algorithm is therefore judged unjustified. The Bayesian method for cloud detection calculates a probability that an observation corresponds to a particular class: clear or cloud. Provided the necessary background information is available, this can be adapted to calculate a probability that an observation corresponds to any number of classes. This was demonstrated here, where the approach was adapted to detect dust, cloud and clear sky simultaneously in a night-time image over sea (generally the most challenging scenario for dust detection). The need for cloud-screening prior to retrieving aerosol observations, which necessarily biases recorded observations of aerosol to those aerosol observations which are spectrally more similar to clear sky than to cloud, is thereby removed for dust. A distribution of simulated Saharan dust observations from another study was used to calculate a PDF, which was made conditional on the pixel NWP Surface Temperature (ST) and Total Column Water Vapour (TCWV). This was combined with the empirical PDF for cloud and the calculated, NWP-conditional, PDF for clear to calculate the normalized posterior probabilities that the pixel observation corresponds to each of the three classes. The latitudeand season-specific prior probabilities required by Bayes Theorem were taken for cloud and clear from International Satellite Cloud Climatology Project (ISCCP) data, and from a dataset of SEVIRI-acquired imagery, for which the Saharan Dust Index (SDI, a measure of the presence of dust) had been calculated, for dust. There being no cloud-clear-dust classified data available for validation, the technique was validated qualitatively through comparison of the three-way classification results against the results of the two-way classification (cloud and clear), and against calculated SDI results (a measure to discriminate between clear and dust). 22 night-time images acquired by the SEVIRI sensor between 2004 and 2006 were used for the validation, and show the technique to produce highly plausible results, although a quantitative assessment is difficult to find. This thesis presents the work undertaken to carry out these developments and extensions to a Bayesian cloud detection scheme. Through this work, several challenges to the technique, such as for example ambiguous classification of pixels around ocean fronts and non-latitude specific prior probabilities of cloud and clear, have been investigated and addressed. The project has extended the range of applications for which the cloud detection technique can be useful to include day-time- and land- imagery applications, in addition to the night-time ocean applications for which it was initially designed. In addition, the work undertaken here has resulted in the method has becoming more physically robust, and more thoroughly validated. A further outcome of this work is the application of the cloud detection technique to the successful classification of imagery into cloud, clear and dust observations, providing a potential solution to areas of NWP and climate research.