大模型训练技术综述
A Survey on Large Model Training Technologies
田海东 1张明政 1常锐 1童贤慧1
作者信息
- 1. 中兴通讯股份有限公司,中国 深圳 518057
- 折叠
摘要
实现高效训练已成为影响大模型应用普及的关键要素之一.按照数据准备、数据加载、模型初始化及评估、训练并行、模型状态保存的一般训练流程,对大模型高效训练的主要技术进行分析和论述.面对大模型规模的持续增长、数据处理类型的扩展,现有大模型训练技术仍存在较大的优化空间.认为未来大模型训练重点研究方向包括以数据为中心、数据加载智能化和异构加速、网络通信领域定制、训练并行及自动化.
Abstract
Achieving efficient training has become one of the key factors affecting the popularization of large model applications.The main technologies of efficient training of large models are analyzed and discussed according to the general training process of data preparation,dataloader,model initialization and evaluation,training parallelism,and model state preservation.In the face of the continuous growth of large model scale and the expansion of data processing types,there is still a large room for optimization of existing large model training tech-nologies.In the future,the key research directions of large model training include data-centric,intelligent dataloader and heterogeneous ac-celeration,customization in the field of network communication,training parallelism and automation.
关键词
大模型/数据准备/数据加载/模型初始化/模型评估/训练并行/训练网络/检查点Key words
large model/data preparation/dataloader/model initialization/model evaluation/training parallelism/training network/checkpoint引用本文复制引用
出版年
2024
NETL
NSTL
万方数据