首页|Low rank optimization for efficient deep learning:making a balance between compact architecture and fast training

Low rank optimization for efficient deep learning:making a balance between compact architecture and fast training

扫码查看
原文链接
  • NETL
  • NSTL
  • 万方数据
Deep neural networks(DNNs)have achieved great success in many data processing applications.However,high computational complexity and storage cost make deep learning difficult to be used on resource-constrained devices,and it is not environmental-friendly with much power cost.In this paper,we focus on low-rank optimization for efficient deep learning tech-niques.In the space domain,DNNs are compressed by low rank approximation of the network parameters,which directly reduces the storage requirement with a smaller number of net-work parameters.In the time domain,the network parameters can be trained in a few subspaces,which enables efficient train-ing for fast convergence.The model compression in the spatial domain is summarized into three categories as pre-train,pre-set,and compression-aware methods,respectively.With a series of integrable techniques discussed,such as sparse pruning,quan-tization,and entropy coding,we can ensemble them in an inte-gration framework with lower computational complexity and storage.In addition to summary of recent technical advances,we have two findings for motivating future works.One is that the effective rank,derived from the Shannon entropy of the norma-lized singular values,outperforms other conventional sparse measures such as the ℓ1 norm for network compression.The other is a spatial and temporal balance for tensorized neural net-works.For accelerating the training of tensorized neural net-works,it is crucial to leverage redundancy for both model com-pression and subspace training.

model compressionsubspace trainingeffective ranklow rank tensor optimizationefficient deep learning

OU Xinwei、CHEN Zhangxin、ZHU Ce、LIU Yipeng

展开 >

School of Information and Communication Engineering,University of Electronic Science and Technology of China,Chengdu 611731,China

国家自然科学基金国家自然科学基金国家自然科学基金Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of ChinaMedico-Engineering Cooperation Funds from University of Electronic Science and Technology of China

62171088U19A205262020106011ZYGX2021YGLH215ZYGX2022YGRH005

2024

10.23919/JSEE.2023.000159

系统工程与电子技术(英文版)
中国航天科工防御技术研究院 中国宇航学会 中国系统工程学会 中国系统仿真学会

系统工程与电子技术(英文版)

CSTPCD
影响因子：0.64
ISSN：1004-4132
年,卷(期)：2024.35(3)
  • 1