FPGA-Based Unified Accelerator for Convolutional Neural Network and Vision Transformer
Considering the problem that traditional Field Programmable Gate Array(FPGA)-based Convolutional Neural Network(CNN)accelerators in computer vision are not adapted to Vision Transformer networks,a unified FPGA accelerator for convolutional neural networks and Transformer is proposed.First,a generalized computation mapping method for FPGA is proposed based on the characteristics of convolution and attention mechanisms.Second,a nonlinear and normalized acceleration unit is proposed to provide acceleration support for multiple nonlinear operations in computer vision networks.Then,we implemented the accelerator design on Xilinx XCVU37P FPGA.Experimental results show that the proposed nonlinear acceleration unit improves the throughput while causing only a small accuracy loss.ResNet-50 and ViT-B/16 achieved 589.94 GOPS and 564.76 GOPS performance on the proposed FPGA accelerator.Compared to the GPU implementation,energy efficiency is improved by a factor of 5.19 and 7.17,respectively.Compared with other large FPGA-based designs,the energy efficiency is significantly improved,and the computing efficiency is increased by 8.02%~177.53%compared to other FPGA accelerators.