基于多级空洞金字塔网络的视频指令学习框架
A VIDEO COMMANDS LEARNING FRAMEWORK BASED ON MULTI-STAGE ATROUS PYRAMID NETWORK
朱展模 1陈俊洪 1杨振国 1刘文印1
作者信息
- 1. 广东工业大学计算机学院 广东广州 510006
- 折叠
摘要
为了从未修剪视频中生成操作指令,提出基于多级空洞金字塔网络(MS-APN)的视频指令学习框架.具体来说,使用空洞卷积金字塔模块捕捉视频多尺度动作特征,并采用多级网络结构优化分割结果,将未修剪视频分割成一系列视频片段并抽取动作特征.运用目标检测模型提取物体特征,并将其与动作特征进行融合,输入分类器识别主体和受体物体.通过定义指令四元组生成机器人指令.在MPII Cooking 2数据集上进行了实验,视频动作分割、操作物体分类、操作指令生成的准确率分别达到了 84.1％、76.5％和62.4％,并成功将系统部署到Baxter机器人上进行验证.
Abstract
We propose a video commands learning framework based on multi-stage atrous pyramid network(MS-APN)for generating robot manipulation instructions from untrimmed videos.Specifically,we introduced an atrous convolution pyramid module to capture multi-scale action features and a multi-stage architecture to refine the segmentation results.The untrimmed video was divided into a series of video segments,and action features were extracted.We applied the object detection model to extract the object features,and they were fused with the action features for inputting into two classifiers to recognize the subject and patient object.A command quadruplet was defined to represent robot commands.Experiments conducted on the MPII Cooking 2 dataset show that the accuracy of the action segmentation,object classification,and robot commands generation reach 84.1％,76.5％,62.4％,respectively.And we successfully deploy our system on a Baxter robot for further verifying the effectiveness of our framework.
关键词
视频指令学习/机器人指令生成/动作分割/空洞卷积Key words
Video commands learning/Robot commands generation/Action segmentation/Atrous convolution引用本文复制引用
基金项目
国家自然科学基金(91748107)
广东省基础与应用基础研究基金(2020A1515010616)
广东省引进创新科研团队计划(2014ZT05G157)
广东省科技创新战略专项(pdjh2020a0173)
出版年
2024
国家科技期刊平台
NSTL
万方数据