Weather-Degraded Image Restoration Based on Visual Prompt Learning
Images captured in real-world scenarios often suffer from weather degradations like random occurrences of rain,haze and snow,which may cause detail occlusion and content deterioration,thereby impacting the effectiveness of subsequent advanced computer vision algorithms.Existing methods for weather-degraded image restoration can be categorized into task-specific,task-aligned and all-in-one types.However,the first two types require specific training for different weather degradations and struggle to adapt to the diverse weather conditions encountered in real-world scenes.Although all-in-one methods achieve the competitive performance across adverse weather degradation removal tasks,they also fail to adapt to the unseen weather degradations,resulting in poor generalization performance.To this end,a weather-degraded image restoration algorithm based on visual prompt learning is proposed in this work,which is a novel paradigm that integrates the pre-trained language-image model with the weather degraded-image restoration.Specifically,even text inputs with similar meanings may yield significantly different latent features when processed through the text encoder of contrastive language-image pre-training(CLIP)model.The general expectation of image restoration is to provide a degraded image and have the model generate its corresponding restored image,rather than multiple different reconstructed images.Therefore,directly using text to guide image reconstruction may lead to unstable solution spaces,often failing to meet the general expectation of image restoration.In response,a query prompt constrained network(QPC-Net)is introduced to utilize the text encoder and image encoder from CLIP to directly encode the latent descriptive features of corresponding ground truth based on the given degraded images.These latent features are further embedded into a pre-trained stable diffusion model using the cross-attention mechanism,thereby constraining the reverse sampling process and facilitating the content reconstruction.QPC-Net consists of two image encoders,with one set of parameters frozen and the other set trainable.Moreover,many existing weather-degraded image algorithms primarily train strict pixel-level mappings between the degraded and clean images,lacking the exploration of knowledge for different image restoration tasks.This limitation makes it difficult for these algorithms to learn the corresponding context for the weather-degraded image restoration tasks not covered in the training dataset,thereby struggling to adapt to different restoration tasks.To address this issue,an example prompt guided network(EPG-Net)is developed to utilize the given example images to guide pre-trained stable diffusion model in learning the context knowledge of corresponding restoration tasks,thereby removing the degradations from query images.Additionally,acquiring suitable example images for complex mixed weather-degraded image restoration tasks are challenging;however EPG-Net can learn the context knowledge from multiple sets of example images.In experimental evaluations conducted on eight seen weather degradation datasets and seven unseen datasets,the proposed algorithm demonstrates significant improvements.Specifically,on the seen weather-degraded datasets,it achieves an average improvement of 2.11dB in peak signal-to-noise ratio(PSNR),4.74％in structural similarity(SSIM),41.08％in perceptual image block similarity(LPIPS)and 24.25％in natural image quality evaluator(NIQE)compared to existing algorithm with similar setting.Additionally,on the unseen weather-degraded datasets,it achieves an average improvement of 1.88 dB in PSNR,5.61％in SSIM,21.40％in LPIPS and 29.29％in NIQE.
computer visionvisual prompt learningin-context learningimage restorationdiffusion model
NETL
NSTL
万方数据
