首页|SWG:an architecture for sparse weight gradient computation

SWG:an architecture for sparse weight gradient computation

扫码查看
原文链接
  • 万方数据
  • 维普
On-device training for deep neural networks(DNN)has become a trend due to various user preferences and scenarios.The DNN training process consists of three phases,feedforward(FF),backprop-agation(BP),and weight gradient(WG)update.WG takes about one-third of the computation in the whole training process.Current training accelerators usually ignore the special computation property of WG and process it in a way similar to FF/BP.Besides,the extensive data sparsity existing in WG,which brings opportunities to save computation,is not well explored.Nevertheless,exploiting the optimization opportunities would meet three underutilization problems,which are caused by(1)the mismatch between WG data dimensions and hardware parallelism,(2)the full sparsity,i.e.,the sparsity of feature map(Fmap),error map(Emap),and gradient,and(3)the workload imbalance resulting from irregular sparsity.In this paper,we propose a specific architecture for sparse weight gradient(SWG)computation.The architecture is designed based on hierarchical unrolling and sparsity-aware(HUSA)dataflow to exploit the optimization opportunities of the special computation property and full data sparsity.In HUSA dataflow,the data di-mensions are unrolled hierarchically on the hardware architecture.A valid-data trace(VDT)mechanism is embedded in the dataflow to avoid the underutilization caused by the two-sided input sparsity.The gradient is unrolled in PE to alleviate the underutilization induced by output sparsity while maintaining the data reuse opportunities.Besides,we design an intra-and inter-column balancer(IIBLC)to dynamically tackle the workload imbalance problem resulting from the irregular sparsity.Experimental results show that with HUSA dataflow exploiting the full sparsity,SWG achieves a speedup of 12.23× over state-of-the-art gradient computation architecture,TrainWare.SWG helps to improve the energy efficiency of the state-of-the-art training accelerator LNPU from 7.56 to 10.58 TOPS/W:

CNNtraininggradient computationsparsityarchitecture

Weiwei WU、Fengbin TU、Xiangyu LI、Shaojun WEI、Shouyi YIN

展开 >

School of Integrated Circuits,Tsinghua University,Beijing 100084,China

Department of Electronic and Computer Engineering,The Hong Kong University of Science and Technology,Hong Kong 999077,China

National Natural Science Foundation of ChinaNational Natural Science Foundation of ChinaNational Natural Science Foundation of ChinaNational Key Research and Development ProgramScience and Technology Innovation 2030-New Generation of AI ProjectBeijing National Research Center for Information Science and TechnologyBeijing Advanced Innovation Center for Integrated Circuits

U19B204162125403921643012021ZD01144002022ZD0115201

2024

10.1007/s11432-022-3807-9

中国科学:信息科学(英文版)
中国科学院

中国科学:信息科学(英文版)

CSTPCDEI
影响因子：0.715
ISSN：1674-733X
年,卷(期)：2024.67(2)
  • 25