SimCSE框架仅使用分类令牌[CLS]token作为文本向量,同时忽略基座模型内层级信息,导致对基座模型输出语义特征提取不充分.本文基于SimCSE框架提出一种融合预训练模型层级特征方法SimCSE-HFF(SimCSE with hierarchical feature fusion,SimCSE-HFF).SimCSE-HFF基于双路并行网络,使用短路径和长路径强化特征学习,短路径使用卷积神经网络学习文本局部特征并进行降维,长路径使用双向门控循环神经网络学习深度语义信息,同时在长路径中利用自编码器融合基座模型内部其他层特征,解决模型对输出特征提取不充分的问题.在STS-B的中文与英文数据集上,SimCSE-HFF方法效果在语义相似度Spearman和Pearson相关性指标上优于传统方法,在不同预训练模型上均得到提升;在下游任务检索问答上也优于SimCSE框架,具有更优秀的通用性.
Text Matching Based on SimCSE Framework Fused with Pre-trained Model Internal Hierarchical Features
The simple contrastive learning of sentence embedding(SimCSE)framework only uses the classification[CLS]tokens as text vectors,and it also neglects the hierarchical information within the base model,which results in insufficient extraction of semantic features from the base model output.Based on the SimCSE framework,this study proposes a method that fuses hierarchical features of pre-trained models,SimCSE with hierarchical feature fusion(SimCSE-HFF).SimCSE-HFF is based on a dual-path parallel network,using short and long paths to strengthen feature learning.The short path uses a convolutional neural network to learn local text features and perform dimensionality reduction,while the long path uses a bidirectional gated recurrent neural network to learn deep semantic information.Additionally,in the long path,an autoencoder is used to fuse features from other layers within the base model,solving the problem of insufficient extraction of output features by the model.On the Chinese and English datasets of spring tools suite-bundle(STS-B),the SimCSE-HFF method outperforms traditional methods in terms of semantic similarity Spearman and Pearson correlation metrics,showing improvements on different pre-trained models.Additionally,it also outperforms the SimCSE framework in downstream task retrieval-based question answering,demonstrating better versatility.
text matchingSimCSEfeature fusionautoencoderparallel network
NETL
NSTL
万方数据
