首页|基于语言-视觉对比学习的多模态视频行为识别方法

基于语言-视觉对比学习的多模态视频行为识别方法

扫码查看
原文链接
  • 万方数据
  • 维普
以对比语言-图像预训练(Contrastive language-image pre-training,CLIP)模型为基础,提出一种面向视频行为识别的多模态模型,该模型从视觉编码器的时序建模和行为类别语言描述的提示学习两个方面对CLIP模型进行拓展,可更好地学习多模态视频表达.具体地,在视觉编码器中设计虚拟帧交互模块(Virtual-frame interaction module,VIM),首先,由视频采样帧的类别分词做线性变换得到虚拟帧分词;然后,对其进行基于时序卷积和虚拟帧分词移位的时序建模操作,有效建模视频中的时空变化信息;最后,在语言分支上设计视觉强化提示模块(Visual-reinforcement prompt module,VPM),通过注意力机制融合视觉编码器末端输出的类别分词和视觉分词所带有的视觉信息来获得经过视觉信息强化的语言表达.在4个公开视频数据集上的全监督实验和2个视频数据集上的小样本、零样本实验结果,验证了该多模态模型的有效性和泛化性.
Multi-modal Video Action Recognition Method Based on Language-visual Contrastive Learning
This paper presents a novel multi-modal model for video action recognition,which is built upon the con-trastive language-image pre-training(CLIP)model.The presented model extends the CLIP model in two ways,i.e.,incorporating temporal modeling in the visual encoder and leveraging prompt learning for language descriptions of action classes,to better learn multi-modal video representations.Specifically,we design a virtual-frame interaction module(VIM)within the visual encoder that transforms class tokens of sampled video frames into virtual-frame tokens through linear transformation,and then temporal modeling operations based on temporal convolution and virtual-frame token shift are performed to effectively model the spatio-temporal change information in the video.In the language branch,we propose a visual-reinforcement prompt module(VPM)that leverages an attention mechan-ism to fuse the visual information,carried by the class token and visual token which are both output by the visual encoder,to enhance the language representations.Fully-supervised experiments conducted on four publicly avail-able video datasets,as well as few-shot and zero-shot experiments conducted on two video datasets,demonstrate the effectiveness and generalization capabilities of the proposed multi-modal model.

Video action recognitionlanguage-visual contrastive learningmulti-modal modeltemporal modelingprompt learning

张颖、张冰冰、董微、安峰民、张建新、张强

展开 >

大连民族大学计算机科学与工程学院 大连 116600

大连民族大学机器智能与生物计算研究所 大连 116600

大连理工大学电子信息与电气工程学部 大连 116024

视频行为识别 语言-视觉对比学习 多模态模型 时序建模 提示学习

国家自然科学基金辽宁省应用基础研究计划国家民委中青年英才培养计划Young and Middleaged Talents Program of the National Civil Affairs Commission

619720622023JH2/10130019161972062

2024

10.16383/j.aas.c230159

自动化学报
中国自动化学会 中国科学院自动化研究所

自动化学报

CSTPCD北大核心
影响因子：1.762
ISSN：0254-4156
年,卷(期)：2024.50(2)
  • 62