Method for Cloud Removal from Optical Remote-Sensing Image Based on Latent Diffusion Model
Objective Optical remote-sensing images are widely used in land planning,natural-resource monitoring,disaster response,and other fields owing to their timeliness,large observation range,and clear visual characteristics.However,approximately 70％of optical remote-sensing images are occluded by clouds.Cloud occlusion complicates ground-information extraction,thus severely limiting the application of optical remote-sensing images.Therefore,cloud removal from optical remote-sensing images is necessary in the preprocessing of remote-sensing images.Compared with the conventional method of removing clouds from optical remote-sensing images,cloud removal based on deep learning presents better effect and a higher accuracy,thus mitigating the issues of conventional algorithms.Recently,denoising diffusion probabilistic models(DDPMs)have attracted much attention due to their generation capabilities beyond generative adversarial networks.DDPMs are generative models that can generate high-quality images closely reflecting the distribution of training data and have achieved the best results in terms of image generation,super-resolution,segmentation,and repair.However,they require significant computing resources to perform denoising.By contrast,the latent diffusion model can obtain high-quality images under less demanding computing requirements.Therefore,this study proposes a cloud-removal method based on the latent diffusion model to remove cloud occlusion from optical remote-sensing images and restore their surface information.Methods The cloud-removal method using the hidden diffusion model for optical remote-sensing images proposed herein is outlined as follows:first,a perceptual compression model is used to learn a hidden space on a cloudless remote-sensing image,and a hidden-space perception equivalent to the original pixel space is established.Training the DDPM in a hidden space can reduce the computing requirements and ensure high-quality image generation.Subsequently,a cloudy image is added to the hidden space to guide the diffusion model to generate a cloudless image,and noise estimation is performed using a U-Net-like cross-covariance self-attention noise estimation network(NEUTViT).The NEUTViT includes a jump connection,cross-covariance attention mechanism,and gated linear unit,which can effectively utilize low-level features,significantly reduce the computational burden,improve the nonlinear characterization ability,and achieve more accurate noise estimation.Additionally,the loss of a similar structural constraint is introduced in the forward process to alleviate the randomness of model generation and guide the model to generate cloudless images closer to the source image,thereby achieving a better cloud-removal effect.Results and Discussions First,the possibility of applying the latent diffusion model for removing clouds in optical remote-sensing images is investigated.The proposed method is evaluated on the STGAN and SEN12MS-CR Winter datasets.On the STGAN dataset,the signal-to-noise ratio and structural similarity are 26.706 and 0.759,respectively,which are 9.855 and 0.171 higher than those yielded by the comparison method on average.On the SEN12MS-CR Winter dataset,the signal-to-noise ratio and structural similarity are 28.779 and 0.798,respectively,which are 7.683 and 0.124 higher than those yielded by the comparison method on average.Experimental results show that the proposed method is superior to the comparison method and can remove clouds in optical remote-sensing images more effectively(T ables 1 and 2).The cloud-removed image yielded by this method offers three advantages:1)high color fidelity;2)favorable textural-detail preservation;3)considerable ability to remove shadows caused by clouds(Figs.6 and 7).Second,we discuss the effects of the cross-covariance attention mechanism and gated linear units on a noise-estimation network.Experiments show that the cross-covariance attention not only improves the noise estimation ability of the noise-estimation network but also significantly reduces the computational complexity of the model.The gated linear unit effectively reduces the computational complexity of the network and enhances the cloud-removal ability of the model(Table 3).Finally,a model using a single loss L2 and another model using joint loss(L2+LSSIM)were compared.The model using the joint loss achieved better results.Compared with the single loss L2,the joint loss enhances the global-structure-recovery ability of the model and improves the quality of the cloud-free image generated by the model(Table 4).Conclusions Cloud removal in remote-sensing images is mandatory in the preprocessing of remote-sensing images and has been investigated extensively.This paper proposes a method to remove clouds from optical remote-sensing images using a hidden diffusion model and restore their surface information.In the forward process,a structural-similarity constraint loss is introduced to alleviate the randomness of model generation,and a U-Net-like cross-covariance attention noise estimation network(NEUTViT)is proposed to estimate the noise distribution more accurately.The cloud removal results obtained on two datasets outperform that of other single-image remote sensing cloud removal methods.The LDMCR model proposed herein performs better than other similar methods;however,it presents some limitations.For example,it can not easily reconstruct surface information enshrouded by large,thick clouds and does not use additional data as aid.In the future,we will use auxiliary data(such as SAR images)and combine them with the cloud-removal tasks of large-scale optical remote-sensing images to investigate cloud removal from optical remote-sensing images using the latent diffusion model.
NETL
NSTL
万方数据
