Model Pruning Algorithm Based on Sparse Optimization and Nesterov Momentum Strategy
With the rapid development of deep learning,the number of parameters and computational complexity of models have exploded,which pose challenges for deployment on mobile terminals.Model pruning has become the key to the implementation and application of deep learning models.At present,the pruning method based on regularization usually adopts L2 regularization combined with the importance standard based on the order of magnitude.It is an empirical method lacking theoretical basis,and its accuracy is difficult to guarantee.Inspired by the Proximal gradient method for solving sparse optimization problems,we propose a Prox-NAG optimization method that can directly generate sparse solutions on deep neural networks and a corresponding iterative pruning algorithm is designed.This method is based on L1 regularization and uses Nesterov momentum to solve the optimization problem.It overcomes the dependence of the original regularization pruning method on L2 regularization and order of magnitude standards,and is a natural extension of sparse optimization from traditional machine learning to deep learning.Pruning experiments are conducted on the ResNet series models on the CIFAR10 dataset,and the results show that the Prox-NAG pruning algorithm has improved its performance compared to the original pruning algorithm.
万方数据
维普
