Information Reconstruction of Cloud – Contaminated Multitemporal Images using Patches

Information Reconstruction of Cloud – Contaminated Multitemporal Images using Patches

Muhsina H

PG Scholar, Dept. of ECE

Muslim Association College of Engineering, Trivandrum, India

Miss. Ashmi Das P

Assistant professor , Dept. of ECE

Muslim Association College of Engineering, Trivandrum, India

Abstract Cloud covers, which are generally present in optical remote sensing images. They limit the usage of acquired images and increase the difficulty in data analysis. Thus, information reconstruction of cloud-contaminated images generally plays an important role in image analysis. This paper proposes a novel method to reconstruct cloud- contaminated information in multitemporal remote sensing images. Based on the concept of utilizing temporal correlation of multitemporal images, a patch-based information reconstruction algorithm is used. That spatiotemporally segments a sequence of images into clusters containing several spatially connected components called patches and then clones information from cloud-free and high-similarity patches to their corresponding cloud-contaminated patches. In addition, a seam that passes through homogenous regions is used in information reconstruction to reduce radiometric inconsistency. Information cloning is solved using an optimization process with the determined seam. These processes enable the proposed method to well reconstruct missing information.

Index Terms Cloud removal, image reconstruction, Landsat Enhanced Thematic Mapper Plus (ETM+), Poisson equation..

1. INTRODUCTION

   The primary limitation of passive remote sensing sensors is their sensitivity to weather conditions during data acquisition. Land scenes are, on average, approximately 35% cloud covered globally [1], significantly reducing the availability of cloud-free surface observations. Clouds in remote sensing images can be regarded as information for measuring liquid water or as contaminations that partially obstruct observation of landscapes. This paper addresses the latter issue in which clouds obstruct land covers, thereby resulting in missing data for passive image sensors. Data analysis, such as classification of land covers, generally requires a cloud-free image composed of several patches that

   are acquired at different times and at different conditions, such as atmospheric conditions, soil moisture, and vegetation

   phenology. These conditions cause the relations between landcover classes and pixel intensities to vary over a data acquisition period. Thus, the approach of replacing the

   cloud-contaminated pixels with their corresponding cloud- free pixels and then linearly adjusting the intensity values of the replaced pixels has been proven inappropriate when the conditions of data acquisition significantly change [2]. Lin et al. [2] proposed a nonlinear scheme instead of a linear one to mathematically formulate the reconstruction problem as a Poisson equation and then solve the equation using a global optimization process. In addition, instead of reconstructing information pixel by pixel [3], [4], which may contain radiometric inconsistency, Lin et al. proposed a patch-based scheme to ease this inconsistency problem. Although this method can yield good cloud-free results, it is sensitive to boundary conditions when solving the Poisson equation and also sensitive to the quality of the selected patch.

   To address the problems above, this paper propose a seam determination approach to select a seam passing through homogenous regions for providing good boundary conditions in reconstruction optimization. Moreover, by utilizing temporal correlation, a clustering algorithm is proposed to segment a cloud-contaminated region into several clusters with similar temporal intensity variations. This segmentation enables the proposed method to handle clouds in a heterogeneous landscape and to select suitable cloud-free pixels. Compared with cloud removal methods in previous studies, the proposed method can yield better cloud-free images in terms of radiometric accuracy and consistency.

2. OVERVIEW OF THE PROPOSED SCHEME

   Fig. 1 shows the workflow of the proposed information reconstruction scheme, which consists of six main steps, namely, cloud detection, image intensity normalization, multitemporal image segmentation, image quality

   assessment, seam determination and image reconstruction. In the proposed scheme, the method proposed by Hagolle et al. [20] is first adopted to detect clouds in the input images, and then, a simple user interface with functions of selection and erasion is provided to manually refine the detected results. In the subsequent step, an image intensity

   normalization process is performed to achieve consistency in the intensity range of the input images. Thereafter, the pixels of the input images are partitioned into several clusters with similar temporal intensity variations and then sorted according to image similarity and the amount of clouds. In the seam determination step, an optimization process based on dynamic programming is performed to search for an optimal seam for each cloud-contaminated region. The proposed information reconstruction algorithm is then performed to recover the issing information of a cloudcontaminated region by solving a Poisson equation with the obtained seam as boundary conditions.

   Fig. 1.Workflow of the proposed reconstruction method scheme.

   1. Cloud Detection

      Given a cloud-contaminated image, called target image and denoted as IT, and a set of its corresponding images captured

      at the same position but different times, called reference images and denoted as {IR1 , . . . , IRn}, the aim is to remove clouds and cloud shadows and to reconstruct the

      information of missing data in the target image IT using the

      reference images {IR1, . . . , IRn}. In the first step, a semiautomatic approach is adopted to detect clouds and cloud shadows in both the target and reference images. The

      approach presented by Huang et al. [7] is applied, and then,

      a simple user interface is provided to manually refine the detection results. In the automatic detection phase, relying on the physical fact that clouds are bright and cold in the thermal band, a thresholding-based approach is adopted to define the cloud boundaries in the spectraltemperature space. Once the cloud pixels are identified, their shadows are roughly predicted according to the cloud location and the solar illumination direction. The dark and connected components within the neighborhood of the predicted shadows are identified as the shadow components. This approach is simple and can detect most clouds and cloud shadows. To robustly generate a cloud-free and cloud- shadow-free image, a manual refinement is further performed to accurately determine the clouds and cloud shadows. In the proposed system, users are allowed to refine the cloud detection results through an interface with simple selection and erasion operations.

   2. Image Intensity Normalization

      As a preprocessing step, image intensity normalization is performed to achieve consistency in the mean and standard deviation of the intensity values of target and reference im- ages. In this process, each reference image with the mean and standard deviation of intensity values [(IR ), (IR )] is linearly transformed into the target image with those of intensity values [(IT ), (IT )]. The normalization is formulated as follows:

      IR(i,j)= [IR(i,j)-µ(IR)] × (IT)/(IR) + µ(IT) (1)

      The cloud-contaminated pixels are excluded in the calcula- tion of (1), and such image normalization process cn ease the difficulty caused by inconsistent intensity dynamic ranges of the input images. The dynamic range of the reference image is altered to make it closer to that of the target image, consequently reducing the difficulty in performing the subsequent steps of information reconstruction. Note that a biased correction may happen since the cloud-contaminated pixels are excluded in the calculation of image normalization.

   3. Spatiotemporal Segmentation

      To select a suitable guidance vector field and to handle cloud covers in a heterogeneous landscape, spatiotemporal segmentation is proposed to segment the input images into several patches with similar temporal intensity variations. a temporal variation map Mvar is generated first according to pixel intensity variations during a defined period. The temporal variation of pixel (i, j) is defined as the average intensity variation in the sequence of target and reference images and is formulated as follows:

      where NumDay(Ik+1 , Ik ) is a function that returns the number of days between the acquisition dates of two consecutive images Ik+1, Ik used in the calculation. Note that thecloud contaminated pixels are excluded in the calculation. since their information is missing, and the intensity variation is normalized by the number of days between the acquisition dates instead of the number of available images because of the inconsistent positions of cloud- free pixels in the temporal domain.

      In the subsequent step, the temporal variation map is par- titioned into several clusters with similar temporal variations, indicating that each cluster probably belongs to a landscape. Any clustering technique can be used here. For simplicity, the efficient and commonly used k-means clustering method is adopted.

   4. Image Quality Assessment

      After clustering, the pixels of target and reference images are grouped into several spatial fragmented clusters, and each cluster contains several connected components called patches. For each patch within the cloud-contaminated region, a cloud- free high-similarity patch is selected from the reference image by utilizing quality assessment. In this paper, the root mean square error (RMSE) is used to estimate patch quality and to select a cloning patch from the reference images.

      To consider heterogeneous landscapes, the RMSE index between the target and reference images is calculated only for the pixels that belong to a cluster. In this manner, a suitable cloning patch can be selected by using only the pixels that belong to the cluster of this patch in quality estimation. In addition, the CCR is considered in patch selection. If the CCR is greater than a defined threshold (set to 80% for all experiments), the reference image will not be selected as a candidate in information reconstruction.

   5. Seam Determination

      In this paper, instead of using the detected cloud boundary [2], a seam that passes though homogenous regions is used. It is because nonhomogeneous regions containing high-gradient content generally have low consistency in pixel intensity and gradient of multitemporal images. Using a seam passing through nonhomogeneous regions as boundary conditions in reconstruction may result in inaccurate inward interpolation and information reconstruction.

      The proposed seam determination approach consists of three steps: search space determintion, cost map generation, optimal seam determination. The search space is determined by first applying the dilation operator to the cloud contaminated region and then applying the erosion

      operator to the dilated region using a 3 × 3 structure element. The eroded region is defined as the search space. The number of dilations and erosions is set to 10,

      indicating that the search space has a width of

      approximately 10 pixels. Cloud contaminated pixels are excluded from the search space, and the cloning patches for the search space are selected using the quality assessment.

   6. Image Reconstruction

While the method proposed by Lin et al. [2] can yield goodresults for most cases, it should be noted that the process of interpolating inward prorogate errors from the boundaries to the cloud-contaminated region may lead to an unnatural result, particularly for a reconstruction with a high-cost seam. To solve such problem, a pixel intensity constraint is included in the optimization to balance fitting

the guidance vector field V and linearly replacing pixel intensity according to that of the selected cloning patch. Including the intensity constraint, the optimization in (1) is reformulated as follows

cp

cp

where fcp is the intensity function defined over the selected cloning patch, and f, is the intensity adjustment of fcp .

where = (2 /x2 )+ (2 /y2 ) is the Laplacian operator, and divV = (v1 /x )+ (v2 /y ) represents divergence of the vector field V = (v1 , v2 ). The cloud-contaminated region in the target image is denoted by , and the cloud boundary (or called seam) is denoted by . Let f be an unknown image intensity function defined over , f be the function defined over the target image minus , and V be a vector field defined as the gradient of selected patches in reference images for guiding information reconstruction.

C. Results and Discussion

Input target image

Seam determination

Cloud detected target image

Reconstructed image

IV. CONCLUSION

In this paper, a novel information reconstruction method for cloud-contaminated images was introduced. Using temporal correlation, multitemporal images were segmented into several patches that have similar temporal variations. Patches in the reference images were then sorted using the RMSE index to select cloning patches, and information of selected patches was seamlessly cloned to corresponding cloud-contaminated patches. The multipatch information reconstruction was solved using an optimization process with the optimal seam. These processes enable the proposed method to well reconstruct information of cloud contaminated regions. The major improvement is that our method makes better use of appropriate spatiotemporal information to reconstruct information, and thus, our method can potentially yield better results in terms of radiometric accuracy and consistency compared with related methods.

REFERENCES

1. J. Ju and D. P. Roy, The availability of cloud-free Landsat ETM+ data over the conterminous United States and globally, Remote Sens. Environ., vol. 112, no. 3, pp. 11961211, Mar. 2008.

2. C. Lin, P. Tsai, K. Lai, and J. Chen, Cloud removal from multitemporal satellite images using information cloning, IEEE Trans. Geosci. Remote Sens., vol. 51, no. 1, pp. 232241, Jan. 2013.

3. F. Melgani, Contextual reconstruction of cloud-contaminated multitem- poral multispectral images, IEEE Trans. Geosci. Remote Sens., vol. 44, no. 2, pp. 442455, Feb. 2006.

4. S. Benabdelkader and F. Melgani, Contextual spatiospectral postreconstruction of cloud-contaminated images, IEEE Geosci. Remote Sens. Lett., vol. 5, no. 2, pp. 204208, Apr. 2008.

5. A. Maalouf, P. Carré, B. Augereau, and C. Fernandez-Maloigne, A bandlet-based inpainting technique for clouds removal from remotely sensed images, IEEE Trans. Geosci. Remote Sens., vol. 47, no. 7, pp. 23632371, Jul. 2009.

6. L. Lorenzi, F. Melgani, and G. Mercier, Inpainting strategies for recon- struction of missing data in VHR images, IEEE Geosci. Remote Sens. Lett., vol. 8, no. 5, pp. 914918, Sep. 2011.

7. C. Huang, N. Thomas, S. N. Goward, J. G. Masek, Z. Zhu, J. R. G. Townshend, and J. E. Vogelmann, Automated masking of cloud and cloud shadow for forest change analysis using Landsat images, Int. J.Remote Sens., vol. 31, no. 20, pp. 54495464, Jun. 2010.