首页|联合多头数据增强与多粒度语义挖掘的图像情感分析

联合多头数据增强与多粒度语义挖掘的图像情感分析

扫码查看
原文链接
  • NETL
  • NSTL
  • 万方数据
图像情感分析是机器视觉领域热点问题，然而情感判断主观性较强，仅分析完整图像难以准确刻画图像中情感语义，且高质量图像情感数据不足。为此，提出联合多头数据增强与多粒度语义挖掘的图像情感分析模型M2。首先，设计多头数据增强方法，基于自动数据增强与主动样本精选策略构建递进式数据增强模型，从"质"与"量"两个角度提升数据集;其次，引入情感区域检测模型完成情感区域增强，深入挖掘图像中情感语义强烈的局部区域，进而联合局部区域与整幅图像构建多粒度图像;然后，基于深度互学习框架及局部区域完成模型预训练，充分挖掘异构SENet网络之间互补的情感语义，并以迁移学习方式指导多粒度图像情感分析;最后，设计自适应特征融合模块，融合异构SENet特征以完成多粒度语义挖掘，实现图像情感分析。在Twitter I和FI数据集上验证M2模型，其准确率分别达到90。97％和81。14％，优于主流基线。M2拥有泛化性更强的数据增强策略，可以为其训练提供坚实的数据基础，且对应的实证分析效果较好，模型具备一定的实用价值。
Image sentiment analysis via multi-head data augmentation and multi-granularity semantics mining
Image sentiment analysis is a hot topic in the computer vision field.However,it is hard to accurately characterize the sentiment semantics by only analyzing the whole image since sentiment judgment is usually subjective.And the image samples with high-quality are scarce.To alleviate the two issues,we propose a multi-head data augment and multi-granularity semantics mining(M2)model.First,a progressive data augmentation model is constructed based on automatic data augmentation and active sample refinement.We improve datasets from the perspectives of quality and quantity.Second,an affective region detection model is introduced for sentiment region augmentation.Intense sentiment semantics is deeply mined from these affective local regions.Then we combine local regions with the whole images to create multi-granularity image data.Third,we pretrain the model through the deep mutual learning framework and affective local regions.The complementary sentiment semantics between heterogeneous SENet networks are fully mined,which is transferred in turn to guide the sentiment analysis of multi-granularity images.Finally,an adaptive feature fusion module is proposed to fuse heterogeneous SENet features to complete multi-granularity semantics mining as well as realize image sentiment analysis.The accuracies of M2 are 90.97％and 81.14％on Twitter I and FI,respectively,which outperform mainstream baselines.The M2 contains a data augmentation strategy with powerful generalization ability,which builds a firm data basis for training.Meanwhile,the corresponding empirical analysis is satisfactory,indicating a certain practicality.

multi-head data augmentationmulti-granularity semantics miningimage sentiment analysisaffective region detectiondeep mutual learningSENet

张红斌、侯婧怡、石皞炜、吕敬钦、李雄、李广丽

展开 >

华东交通大学软件学院,南昌 330013

华东交通大学信息工程学院,南昌 330013

多头数据增强 多粒度语义挖掘 图像情感分析 情感区域检测 深度互学习 SENet

国家自然科学基金国家自然科学基金江西省自然科学基金江西省自然科学基金江西省主要学科学术和技术带头人培养计划江西省社会科学基金江西省重点研发计划重点项目

621610116236102720232BAB20200420202BABL20204420204BCJL2303522TQ0120223BBE51036

2024

10.13195/j.kzyjc.2022.1807

控制与决策
东北大学

控制与决策

CSTPCD北大核心
影响因子：1.227
ISSN：1001-0920
年,卷(期)：2024.39(6)