Abstract
Deep neural networks(DNNs)have achieved great success in many data processing applications.However,high computational complexity and storage cost make deep learning difficult to be used on resource-constrained devices,and it is not environmental-friendly with much power cost.In this paper,we focus on low-rank optimization for efficient deep learning tech-niques.In the space domain,DNNs are compressed by low rank approximation of the network parameters,which directly reduces the storage requirement with a smaller number of net-work parameters.In the time domain,the network parameters can be trained in a few subspaces,which enables efficient train-ing for fast convergence.The model compression in the spatial domain is summarized into three categories as pre-train,pre-set,and compression-aware methods,respectively.With a series of integrable techniques discussed,such as sparse pruning,quan-tization,and entropy coding,we can ensemble them in an inte-gration framework with lower computational complexity and storage.In addition to summary of recent technical advances,we have two findings for motivating future works.One is that the effective rank,derived from the Shannon entropy of the norma-lized singular values,outperforms other conventional sparse measures such as the ℓ1 norm for network compression.The other is a spatial and temporal balance for tensorized neural net-works.For accelerating the training of tensorized neural net-works,it is crucial to leverage redundancy for both model com-pression and subspace training.
基金项目
国家自然科学基金(62171088)
国家自然科学基金(U19A2052)
国家自然科学基金(62020106011)
Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China(ZYGX2021YGLH215)
Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China(ZYGX2022YGRH005)
NETL
NSTL
万方数据