非语言信息增强和对比学习的多模态情感分析模型
A Multimodal Sentiment Analysis Model Enhanced with Non-verbal Information and Contrastive Learning
刘佳 1宋泓 2陈大鹏 1王斌 2张增伟2
作者信息
- 1. 南京信息工程大学天长研究院 滁州 239356;南京信息工程大学自动化学院 南京 210044;江苏省智能气象探测机器人工程研究中心 南京 210044;江苏省大气环境与装备技术协同创新中心 南京 210044
- 2. 南京信息工程大学天长研究院 滁州 239356;南京信息工程大学自动化学院 南京 210044
- 折叠
摘要
因具有突出的表征和融合能力,深度学习方法近年来越来越多地被应用于多模态情感分析领域.已有的研究大多利用文字、面部表情、语音语调等多模态信息对人物的情绪进行分析,并主要使用复杂的融合方法.然而,现有模型在长时间序列中未充分考虑情感的动态变化,导致情感分析性能不佳.针对这一问题,该文提出非语言信息增强和对比学习的多模态情感分析网络模型.首先,使用长程文本信息去促使模型学习音频和视频在长时间序列中的动态变化,然后,通过门控机制消除模态间的冗余信息和语义歧义.最后,使用对比学习加强模态间的交互,提升模型的泛化性.实验结果表明,在数据集CMU-MOSI上,该模型将皮尔逊相关系数(Corr)和F1值分别提高了3.7%和2.1%;而在数据集CMU-MOSEI上,该模型将"Corr"和"F1值"分别提高了1.4%和1.1%.因此,该文提出的模型可以有效利用模态间的交互信息,并去除信息冗余.
Abstract
Deep learning methods have gained popularity in multimodal sentiment analysis due to their impressive representation and fusion capabilities in recent years.Existing studies often analyze the emotions of individuals using multimodal information such as text,facial expressions,and speech intonation,primarily employing complex fusion methods.However,existing models inadequately consider the dynamic changes in emotions over long time sequences,resulting in suboptimal performance in sentiment analysis.In response to this issue,a Multimodal Sentiment Analysis Model Enhanced with Non-verbal Information and Contrastive Learning is proposed in this paper.Firstly,the paper employs long-term textual information to enable the model to learn dynamic changes in audio and video across extended time sequences.Subsequently,a gating mechanism is employed to eliminate redundant information and semantic ambiguity between modalities.Finally,contrastive learning is applied to strengthen the interaction between modalities,enhancing the model's generalization.Experimental results demonstrate that on the CMU-MOSI dataset,the model improves the Pearson Correlation coefficient(Corr)and F1 score by 3.7%and 2.1%,respectively.On the CMU-MOSEI dataset,the model increases"Corr"and"F1 score"by 1.4%and 1.1%,respectively.Therefore,the proposed model effectively utilizes intermodal interaction information while eliminating information redundancy.
关键词
多模态情感分析/多模态融合/信息增强/多层感知器Key words
Multimodal emotion analysis/Multimodal fusion/Information enhancement/MultiLayer Perceptron(MLP)引用本文复制引用
基金项目
国家自然科学基金(61773219)
国家自然科学基金(62003169)
江苏产业前瞻与关键技术重点项目(BE2020006-2)
江苏省自然科学基金青年基金(BK20200823)
出版年
2024
NETL
NSTL
万方数据