基于时间动态帧选择与时空图卷积的可解释骨架行为识别
Temporal dynamic frame selection and spatio-temporal graph convolution for interpretable skeleton-based action recognition
梁成武 1杨杰 1胡伟 1蒋松琪 1钱其扬 2侯宁2
作者信息
- 1. 三峡大学电气与新能源学院,湖北 宜昌 443002;河南城建学院电气与控制工程学院,河南 平顶山 467036
- 2. 河南城建学院电气与控制工程学院,河南 平顶山 467036
- 折叠
摘要
骨架行为识别是计算机视觉和机器学习领域的研究热点.现有数据驱动型神经网络往往忽略骨架序列时间动态帧选择和模型内在人类可理解的决策逻辑,造成可解释性不足.为此提出一种基于时间动态帧选择与时空图卷积的可解释骨架行为识别方法,以提高模型的可解释性和识别性能.首先利用骨架帧置信度评价函数删除低质骨架帧,以解决骨架序列噪声问题.其次基于人体运动领域知识,提出自适应时间动态帧选择模块用于计算运动行为显著区域,以捕捉关键人体运动骨架帧的动态规律.为学习行为骨架节点内在拓扑结构,改进时空图卷积网络用于可解释骨架行为识别.在NTU RGB+D,NTU RGB+D 120 和FineGym这 3 个大型公开数据集上的实验评估表明,该方法的骨架行为识别准确率优于对比方法并具有可解释性.
Abstract
Skeleton-based action recognition is a prominent research topic in computer vision and machine learning.Existing data-driven neural networks often overlook the temporal dynamic frame selection of skeleton sequences and lack the understandable decision logic inherent in the model,resulting in insufficient interpretability.To this end,we proposed an interpretable skeleton-based action recognition method based on temporal dynamic frame selection and spatio-temporal graph convolution,thereby enhancing the interpretability and recognition performance.Firstly,the quality of skeleton frames was estimated using the joint confidence to remove low-quality skeleton frames,addressing the skeleton noise problem.Secondly,based on the domain knowledge of human activity,an adaptive temporal dynamic frame selection module was proposed for calculating the motion salient regions to capture the dynamic patterns of key skeleton frames in human motion.To represent the intrinsic topology of human joints,an improved spatiotemporal graph convolutional network was used for interpretable skeleton-based action recognition.Experiments were conducted on three large public datasets,including NTU RGB+D,NTU RGB+D 120,and FineGym,and the results demonstrated that the recognition accuracy of this method outperformed comparative methods and possessed interpretability.
关键词
行为识别/骨架序列/可解释/运动显著区域/时空图卷积网络Key words
action recognition/skeleton sequence/interpretability/motion salient regions/spatio-temporal graph convolutional network引用本文复制引用
基金项目
国家自然科学基金项目(62176086)
国家自然科学基金项目(U1804152)
河南省科技攻关计划项目(242102211055)
出版年
2024
NETL
NSTL
万方数据