计算机研究与发展2024,Vol.61Issue(5) :1206-1217.DOI:10.7544/issn1000-1239.202330960

基于多模态大语言模型的攻击性模因解释生成方法

A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models

林萌 戴程威 郭涛
计算机研究与发展2024,Vol.61Issue(5) :1206-1217.DOI:10.7544/issn1000-1239.202330960

基于多模态大语言模型的攻击性模因解释生成方法

A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models

林萌 1戴程威 1郭涛2
扫码查看

作者信息

  • 1. 中国科学院信息工程研究所 北京 100019;中国科学院大学网络空间安全学院 北京 100049
  • 2. 中国科学院信息工程研究所 北京 100019
  • 折叠

摘要

随着5G的发展,攻击性言论逐渐以多模态的方式在社交网络上广泛传播.因此,攻击性模因的检测与解释生成对于提高内容审核效果、维护和谐健康的舆论场环境有着重要的作用.现有的攻击性模因解释生成研究只关注于攻击对象和攻击内容,忽略了模因包含的社会背景知识和隐喻表达手法,无法全面、准确地解释攻击性模因的含义,大大限制了解释的应用范围.为了应对这一挑战,提出一种基于多模态大模型的攻击性模因解释生成方法,通过增强攻击目标、攻击内容和隐喻识别等多种指令数据,利用其微调多模态大模型,以提升大模型对攻击性模因的解释生成能力.实验结果证实,该方法生成的解释具有3点优势:一是相比基线模型在BERTScore评估指标上提高了 19%;二是解释中包含了攻击性隐喻表达的相关背景知识;三是在处理未见的模因数据时也表现出良好的泛化性能.

Abstract

With the advancement of 5G technology,offensive speech has increasingly proliferated across social networks in the form of multimodal memes.Consequently,the detection and interpretive generation of offensive memes play a crucial role in enhancing content moderation effectiveness and maintaining a harmonious and healthy public discourse environment.Existing studies on the interpretive generation of offensive memes focus solely on the targets and content of offense,neglecting the societal background knowledge and metaphorical expressions embedded in memes.This oversight limits the ability to comprehensively and accurately interpret the meaning of offensive memes,thus constraining the applicability of interpretations.To address this challenge,we propose a method based on multimodal large language model for generating interpretations of offensive memes.By augmenting elements such as offense targets,the content of the offense,and metaphor recognition into the instruction tuning data,we can effectively improve the multimodal large model's proficiency in interpretively generating offensive memes through instruction tuning.The experimental outcomes validate three key strengths of our method:first,it achieves a notable 19%enhancement in the BERTScore evaluation metric over baseline models;second,it incorporates comprehensive background knowledge pertinent to offensive metaphorical expressions within its interpretations;third,it exhibits strong generalization capabilities when handling previously unseen meme data.

关键词

攻击性模因/解释生成/多模态大语言模型/数据增强/指令微调

Key words

offensive meme/explanation generation/multi-modal LLM/data augment/instruction fine-tuning

引用本文复制引用

出版年

2024
计算机研究与发展
中国科学院计算技术研究所 中国计算机学会

计算机研究与发展

CSTPCD北大核心
影响因子:2.649
ISSN:1000-1239
参考文献量43
段落导航相关论文