首页|融合交叉注意力的突发事件多模态中文反讽识别模型

融合交叉注意力的突发事件多模态中文反讽识别模型

扫码查看
原文链接
  • 国家科技期刊平台
  • NETL
  • NSTL
  • 万方数据
网民在社交媒体参与突发事件讨论时,时常会采用反讽修辞方式表达对事件的看法,此举导致情感分析的难度增加,且已有中文反讽识别对社交媒体中网民发布的多模态评论研究较少,有必要对图文多模态中文反讽识别进行深入研究.运用交叉注意力机制捕捉模态间的不一致性表达,提出融合交叉注意力的多模态中文反讽识别模型(fuse cross attention model,FCAM).在模型中,首先运用TextCNN(text convolutional neural net-works)和ResNet(deep residual network)分别提取中文文本浅层特征和图像特征,再运用交叉注意力机制分别得到文本层和图像层的注意力特征,按照残差方式分别实现文本浅层特征和文本层注意力特征的连接、图像特征和图像层注意力特征的连接,使用注意力机制融合 2 个特征表示,经过分类层得到反讽分类结果.基于某一地区新冠疫情期间相关话题的微博评论数据,构建出突发公共卫生事件多模态中文反讽数据集,在该数据集上试验验证,相较于基准模型,FCAM具有一定的优越性.
A multimodal Chinese sarcasm detection model for emergencies based on cross attention
Internet users often use sarcasm when discussing emergencies on social media,which complicates emotional analysis.In addition,there is a lack of research on multimodal comments,particularly those in Chinese,and their use of sarcasm on social media platforms.Therefore,it is necessary to delve deeper into sarcasm detection in multimodal Chinese content,specifically within images and text.To address this need,we propose a multimodal Chinese sarcasm detection model called the fuse cross-attention model(FCAM).This model incorporates a cross-attention mechanism to identify inconsistencies between modes.The text convolutional neural network(TextCNN)is used to extract basic fea-tures of Chinese text,while the deep residential network(ResNet)is used to extract image features.The cross-attention mechanism is used to obtain attention features from the text and image layers.The residual method is employed to estab-lish a connection between the basic text features and the text layer's attention features,as well as a link between the im-age features and the image layer's attention features.These two feature representations are fused using the attention mechanism,resulting in the sarcasm classification results through the classification layer.We have constructed a mul-timodal Chinese sarcasm data set based on Weibo comment data related to the COVID-19 pandemic in a specific region.Experimental testing on this data set confirms that FCAM holds certain advantages over the benchmark model.

emergencysocial mediamultimodal commentChinese sarcasm detectionChinese sarcasm datasetcross-attention mechanismattention mechanismsentiment analysis

胡文彬、陈龙、黄贤波、陈晨、仲兆满

展开 >

江苏海洋大学 计算机工程学院, 江苏 连云港 222005

江苏省海洋资源开发研究院, 江苏 连云港 222005

突发事件 社交媒体 多模态评论 中文反讽识别 中文反讽数据集 交叉注意力机制 注意力机制 情感分析

国家自然科学基金江苏省高等学校"青蓝工程"优秀教学团队项目

721740792022-29

2024

10.11992/tis.202212011

智能系统学报
中国人工智能学会　哈尔滨工程大学

智能系统学报

CSTPCD北大核心
影响因子：0.672
ISSN：1673-4785
年,卷(期)：2024.19(2)
  • 27