面向Vision Transformer模型的剪枝技术研究
Research on Pruning Technology for Vision Transformer Model
查秉坤 1李朋阳 1陈小柏1
作者信息
- 1. 南京邮电大学计算机学院软件学院网络空间安全学院,江苏南京 210023
- 折叠
摘要
本文针对Vision Transformer(ViT)模型开展剪枝技术研究,探索了多头自注意力机制中的QKV(Query、Key、Value)权重和全连接层(Fully Connected,FC)权重的剪枝问题.针对ViT模型本文提出了3组剪枝方案:只对QKV剪枝、只对FC剪枝以及对QKV和FC同时进行剪枝,以探究不同剪枝策略对ViT模型准确率和模型参数压缩率的影响.本文开展的研究工作为深度学习模型的压缩和优化提供了重要参考,对于实际应用中的模型精简和性能优化具有指导意义.
Abstract
This article focuses on the pruning technology research of the Vision Transformer(ViT)model,exploring the pruning problem of QKV(Query,Key,Value)weights and Fully Connected(FC)weights in the multi head self attention mechanism.This article proposes three pruning schemes for the ViT model:QKV pruning only,FC pruning only,and simultaneous pruning of QKV and FC to explore the effects of different pruning strategies on the accuracy and parameter compression of the ViT model.The research conducted in this article provides important references for the compression and optimization of deep learning models,and has guiding significance for model simplification and performance optimization in practical applications.
关键词
Vision/Transformer模型/剪枝/准确率Key words
Vision Transformer model/pruning/accuracy引用本文复制引用
出版年
2024
NETL
NSTL
万方数据