针对当前方法普遍存在较为严重的细节结构信息丢失与事件间重叠的问题,提出一种基于双向特征金字塔的密集视频描述生成方法(dense video captioning with bilateral feature pyramid net,BFPVC).BFPVC通过带有自底向上、自顶向下、横向链接3条分支的双向特征金字塔强化视频多尺度特征图,兼顾对时序信息、空间信息、语义信息的特征表示,解码器从强化后的视频特征中捕获更加全面的事件候选集,从而为对应的视频事件生成更加丰富、详尽的文本描述.在ActivityNet Captions数据集和YouCook2数据集上的实验结果表明,BFPVC与同类模型相比生成的文本描述更详细、丰富,验证了双向特征金字塔在密集视频描述领域的有效性.
Dense video captioning with bilateral feature pyramid net
In order to solve the generally existing problem of detail structure information loss and inter-event overlap in current dense video description generation method,an end-to-end dense video captioning with bilateral feature pyramid net(BFPVC)was pro-posed.BFPVC strengthened the multi-scale feature maps of videos by incorporating a bidirectional feature pyramid with three branches of bottom-up,top-down,and lateral connections.It balanced the feature representation of temporal,spatial,and semantic information.The backbone network captured a more comprehensive set of event candidates from the enhanced video features,thereby generating richer and more detailed captions for the corresponding video events.The results of this experiments on the ActivityNet Captions dataset and YouCook2 dataset demonstrate that the BFPVC method generates more detailed and richer cap-tions than similar frameworks,validating the effectiveness of the bilateral feature pyramid net in the field of dense video captioning.
dense video captioningvideo captioningvideo understandingfeature pyramid netnatural language processing
NETL
NSTL
万方数据
