Deep learning-based spectral image super-resolution:a survey
The goal of spectral image super-resolution technology is to recover images with high spatial resolution and spec-tral resolution from images with low spatial resolution and spectral resolution.Images of high spatial and spectral resolution are widely used in remote sensing fields such as vegetation survey,geological exploration,environmental protection,anomaly detection,and target tracking.With the rise of deep learning,spectral image super-resolution algorithms based on deep learning have emerged.In particular,the emergence of technologies such as end-to-end neural networks,generative adversarial networks,and deep unfolding networks has made a qualitative leap in spectral image super-resolution perfor-mance.This study comprehensively discusses and analyzes cutting-edge deep learning algorithms under different spectral image super-resolution task scenarios.First,we introduce the basic concepts of spectral image super-resolution and the definitions of different super-resolution scenarios.Focusing on the two major scenarios of single-image super-resolution and fusion super-resolution,the basic concepts of various methods are elaborated from multiple perspectives such as super-resolution dimensions,super-resolution data types,basic frameworks,and supervision methods,and their characteristics are discussed.Second,this study summarizes the limitations of various algorithms and proposes directions for further improvement.Furthermore,the commonly used datasets in different fusion scenarios are briefly introduced,and the spe-cific definitions of various evaluation indicators are clarified.For each super-resolution task,this study comprehensively compares the performance of representative algorithms from multiple perspectives such as qualitative evaluation and quanti-tative evaluation.Finally,this study summarizes the research results and discusses some serious challenges faced in the field of spectral image super-resolution,while also looking forward to possible future research directions.First,from the perspective of super-resolution scenarios,the existing spectral image super-resolution algorithms can be divided into two categories,namely,single image super-resolution and fusion-based super-resolution.Specifically,single spectral image super-resolution is designed to generate high-resolution output images from a single low-resolution input image.According to the direction of super-resolution,single image super-resolution can be divided into spatial super-resolution,spectral super-resolution,and spatial-spectral super-resolution.Fusion-based spectral image super-resolution is designed to fuse images of different modes into a single image with high spatial and spectral resolution.According to the different modes of fusion images,fusion-based spectral image super-resolution can be divided into pansharpening and multispectral and hyper-spectral images fusion.Moreover,deep learning-based spectral image super-resolution methods can be categorized into end-to-end neural network based(E2EN-based)spectral image super-resolution framework,generative adversarial network-based(GAN-based)spectral image super-resolution framework,and deep unfolding network-based(DUN-based)spectral image super-resolution framework according to the network architecture.The E2EN-based spectral image super-resolution framework designs various network structures to mine nonlinear mapping relationships between low-resolution and high-resolution images.According to the basic computing unit of network structure,it can be divided into convolu-tional neural network-based method and Transformer-based method.The GAN-based spectral image super-resolution frame-work realizes the spectral image super-resolution through the game between the generator and the discriminator.The DUN-based spectral image super-resolution framework combines traditional optimization algorithms and deep learning,and it unfolds iterative optimization steps to form deep neural networks.From the perspective of supervision paradigm,the deep learning algorithms can also be classified into unsupervised and supervised categories.The supervised approaches mini-mize the distance between super-resolved spectral image and ground truth,while unsupervised algorithms design loss func-tion through the similarity between super-resolved and input images or through the game of the generator and the discrimina-tor.Our critical review describes the main concepts and characteristics of each approach for different spectral image super-resolution tasks according to the network architecture and supervision paradigm.Second,we introduce the representative datasets and evaluation metrics.We divide the datasets into categories of single spectral image super-resolution datasets and fusion-based spectral image super-resolution datasets.Furthermore,the evaluation metrics can be grouped into full-reference metrics and no-reference metrics.Some full-reference metrics are widely used for the quantitative evaluation of spectral image super-resolution,including peak signal-to-noise,structural similarity,spectral angle mapper,and rela-tive dimensionless global error in synthesis.Third,we provide the quantitative and qualitative experimental results of dif-ferent spectral image super-resolution tasks.Finally,we summarize the challenges and problems in the study of deep learning-based spectral image super-resolution and conduct forecasting analysis,such as high-quality spectral image super-resolution dataset,model-driven and deep learning combined spectral image super-resolution method,real-time spectral image super-resolution,and comprehensive evaluation metrics.The methods and datasets mentioned are linked at https://github.com/ColinTaoZhang/DL-based-spectral-super-resolution.
deep learningsuper-resolutionspectral imagesingle image super-resolutionfusion-based super-resolution
NETL
NSTL
万方数据
