CNN-Transformer结合对比学习的高光谱与LiDAR数据协同分类
Collaborative classification of hyperspectral and LiDAR data based on CNN-transformer
吴海滨 1戴诗语 1王爱丽 1岩堀祐之 2于效宇3
作者信息
- 1. 哈尔滨理工大学 测控技术与通信工程学院 黑龙江省激光光谱技术及应用重点实验室,黑龙江 哈尔滨 150080
- 2. 中部大学 计算机科学学院,日本 爱知 487-8501
- 3. 电子科技大学 中山学院 电子信息学院,广东 中山 528400
- 折叠
摘要
针对高光谱图像(hyperspectral images,HSI)与LiDAR数据多模态分类任务中的跨模态信息表达和特征对齐等问题,提出一种基于对比学习CNN-Transformer高光谱和LiDAR数据协同分类网络(Contrastive Learning based CNN-Transformer Network,CLCT-Net).CLCT-Net通过由ConvNeXt V2 Block构成的共有特征提取模块,获得不同模态间的共性特征,解决异构传感器数据之间语义对齐的问题.构建了包含空间-通道分支和光谱上下文分支的双分支HSI编码器,以及结合频域自注意力机制的LiDAR编码器,以获取更丰富的特征表示.利用集成对比学习进行分类,进一步提升多模态数据协同分类的精度.在 Houston 2013 和 Trento 数据集上的实验结果表明,相较于其他高光谱图像和Li-DAR数据分类模型,本文所提模型获得了更高的地物分类精度,分别达到了92.01%和98.90%,实现了跨模态数据特征的深度挖掘和协同提取.
Abstract
To tackle the challenges in multimodal classification tasks involving hyperspectral images(HSI)and LiDAR data,such as cross-modal information expression and feature alignment,this paper introduces a contrastive learning-based multi-branch CNN-Transformer network(CLCT-Net)for the joint classifica-tion of hyperspectral and LiDAR data.Initially,CLCT-Net employs a feature extraction module with a ConvNeXt V2 Block to capture shared features across different modalities,addressing the semantic align-ment issue between data from heterogeneous sensors.It then develops a dual-branch HSI encoder with spa-tial channel and spectral context branches,alongside a LiDAR encoder enhanced by a frequency domain self-attention mechanism,to secure more comprehensive feature representations.Lastly,it leverages en-semble contrastive learning for classification to further refine the accuracy of multimodal collaborative clas-sification.Experimental evaluations on the Houston 2013 and Trento datasets demonstrate that the pro-posed model excels in extracting and integrating cross-modal data features,achieving superior ground ob-ject classification accuracies of 92.01%and 98.90%,respectively,when compared to existing models for classifying hyperspectral images and LiDAR data.
关键词
高光谱图像/激光雷达数据/Transformer/卷积神经网络/对比学习Key words
hyperspectral image/LiDAR data/transformer/convolutional neural network/contrastive learning引用本文复制引用
基金项目
黑龙江省自然科学基金资助项目(JJ2023LH1143)
黑龙江省重点研发计划资助项目(JD2023SJ19)
"一带一路"创新人才交流外国专家项目(G2022012010L)
黑龙江省级领军人才梯队后备带头人资助项目()
出版年
2024
国家科技期刊平台
NSTL
万方数据