计算机与网络2024,Vol.50Issue(4) :283-288.DOI:10.20149/j.cnki.issn1008-1739.2024.04.001

基于Vision Transformer的时空卷积网络设计

Spatiotemporal Convolutional Network Design Based on Vision Transformer

谢英红 郝岩 韩晓微 高强 阴彪 王朝辉
计算机与网络2024,Vol.50Issue(4) :283-288.DOI:10.20149/j.cnki.issn1008-1739.2024.04.001
✕

基于Vision Transformer的时空卷积网络设计

Spatiotemporal Convolutional Network Design Based on Vision Transformer

谢英红 1郝岩 1韩晓微 1高强 1阴彪 1王朝辉1
扫码查看

作者信息

  • 1. 沈阳大学信息工程学院,辽宁沈阳 110044
  • 折叠

摘要

目前主流人体动作识别大部分都是基于卷积神经网络(Convolutional Neural Network,CNN)实现,而CNN容易忽略视频中的空间位置信息,从而降低了视频空间频域中动作识别能力.同时传统CNN不能快速定位到关键的特征位置,并且在训练过程中不能并行计算导致效率低.为了解决传统CNN在处理时间频域和多并行计算问题,提出了基于视觉Transformer(Vision Transformer,ViT)和 3D 卷积网络学习时空特征(Learning Spatiotemporal Features with 3D Convolutional Network,C3D)的人体动作识别算法.使用C3D提取视频的多维特征图、ViT的特征切片窗口对多维特征进行全局特征分割;使用Transformer的编码-解码模块对视频中人体动作进行预测.实验结果表明,所提的人体动作识别算法在UCF-101、HMDB51数据集上提高了动作识别的准确率.

Abstract

At present,the mainstream human action recognition is mostly based on Convolutional Neural Network(CNN),which tend to ignore the spatial position information in the video,thus reducing the action recognition ability in the spatial frequency domain of video.At the same time,the traditional CNN can not locate the key feature position quickly,and the parallel computation efficiency is low in the training process.In order to solve the problems of processing time-frequency domain and multi-parallel computation in traditional CNN,a human action recognition algorithm based on the Vision Transformer(ViT)and Learning Spatiotemporal Features with 3D Convolutional Network(C3D)is proposed.Firstly,the multi-dimensional feature map of the video is extracted by using C3D.The feature slice window of ViT is used for global feature segmentation of multi-dimensional features.Finally,the coding-decoding module of transformer is used to predict human actions in the video.The experimental results show that the accuracy of action recognition is improved by the human action recognition algorithm on UCF-101 and HMDB51 datasets.

关键词

动作识别/视觉Transformer/卷积神经网络/3D卷积网络学习时空特征/注意力机制

Key words

action recognition/ViT/CNN/C3D/attention mechanism

引用本文复制引用

出版年

2024
计算机与网络
工业和信息化部电子无线通信专业情报网

计算机与网络

CHSSCD
影响因子:0.149
ISSN:1008-1739
段落导航相关论文