一种面向通用计算设备的自动流水线并行训练框架
Automatic Pipeline Parallel Training Framework for General-purpose Computing Devices
钟震宇 1林勇良 1王昊天 1李东闻 1孙羽菲 1张玉志1
作者信息
摘要
训练大规模神经网络通常会出现单个计算节点的内存和计算能力不足的情况,需要通过多个节点分布式训练来实现.现有的分布式深度学习框架主要针对特定的硬件环境设计,不能够有效适应各类通用计算设备.为支持大规模深度神经网络的高效训练,实现了一种通用的自动流水线并行分布式训练框架.本框架通过结合基于流水线并行的模型并行策略与神经网络模型自动拆分算法,实现了在包括国内新一代超级计算机在内的通用计算机集群上,对大规模神经网络模型与训练数据进行自动并行化处理和训练,显著减轻单个计算节点的内存和计算压力.该框架无需人工调整,可以自动高效地在多节点分布式环境中部署深度神经网络,不仅适用于超级计算机等高性能计算机集群,还可以部署到其他通用的分布式计算环境中,为大规模神经网络的自动化分布式训练提供支持.
Abstract
Training large-scale neural networks usually exceeds the memory and computing capacity of a single computing node,which requires distributed training using multiple nodes.Existing distributed deep learning frameworks are mainly designed for specific hardware environments and cannot effectively adapt to various general-purpose computing devices.To support the effi-cient training of large-scale deep neural networks,this paper implements a general-purpose automatic pipeline parallel distributed training framework.This framework combines the model parallel strategy based on pipeline parallelism with the algorithm that automatically splits the neural network model,and realizes the automatic parallelization and training of large-scale neural network models and training data on general computer clusters,including the new generation of supercomputers in China,significantly re-ducing the memory and computing pressure of a single computing node.The framework does not require manual adjustment,and can automatically and efficiently deploy deep neural networks to multi-node distributed environments.It is not only suitable for supercomputers and other high-performance computer clusters,but also can be deployed to other general distributed computing environments,providing support for the automatic distributed training of large-scale neural networks.
关键词
流水线并行/深度神经网络/超级计算机/MPI/并行计算Key words
Pipeline parallelism/Deep neural network/Supercomputer/Message passing interface/Parallel computing引用本文复制引用
出版年
2024
NETL
NSTL
万方数据