Cloud-based intelligent computing center ten-thousand card cluster innovation and practice
扫码查看
点击上方二维码区域，可以放大扫码查看
原文链接
NETL
NSTL
万方数据
为解决智算中心超大规模算力集群算力可用率低、国产技术成熟度低、大规模组网效率存在瓶颈、运营运维复杂等问题,提出了一种基于云计算技术构建智算中心万卡集群的系统.采用18 432块神经网络处理单元(neural processing unit,NPU)卡和优化后的基于以太网的远程直接内存访问(remote direct memory access,RDMA)网络构建云化的智算中心万卡集群,结合软件定义网络(software defined network,SDN)技术实现RDMA网络租户隔离,实现了链路负载均衡误差小于10%,集群All-Reduce带宽达35 GB/s以上.采用优化后的分布式存储协议,实现模型断点恢复时长缩短为原来的1/2.验证结果表明,经过软硬件协同优化,国产化的NPU万卡集群不仅能够满足千亿参数大模型训练的需求,未来更可以支撑万亿参数大模型训练任务.
To address issues such as low availability of computing power in ultra-large scale computing clusters of in-telligent computing centers,low maturity of domestically produced technologies,bottlenecks in large-scale network-ing efficiency,and complex operations and maintenance,a system based on cloud computing technology for construct-ing a ten-thousand card cluster in an intelligent computing center was proposed.A ten-thousand card cluster was con-structed using 18 432 NPU units and an optimized RDMA network.A multi-plane network architecture was adopted,in conjunction with SDN technology to achieve RDMA network tenant isolation.The network load balancing strategy was optimized,resulting in a link load balancing error of less than 10%and an All-Reduce bandwidth of over 35 GB/s.By employing the optimized distributed storage protocol,the model's breakpoint recovery time was reduced to half of its original duration.The validation results demonstrate that the domestic NPU ten-thousand card cluster,with the col-laborative optimization of software and hardware,can not only meet the training needs of large models with hundreds of billions of parameters but also support the training tasks of large models with trillions of parameters.
NETL
NSTL
万方数据
