应用程序属性感知的Yarn资源调度模型研究
Research on application attribute aware Yarn resource scheduling model
陈宁宁1
作者信息
- 1. 西安外事学院工学院,西安 710077
- 折叠
摘要
Hadoop应用程序存在计算密集属性和调度时间属性,但是Hadoop大数据平台集成的第二代资源管理器Yarn内置的三种资源调度器无法将相同属性的应用程序均衡分配到计算节点上,导致部分节点负载过高,出现严重的计算任务长尾效应.文中提出了一种应用程序属性感知的Yarn负载均衡调度模型——APB Scheduler.APB Scheduler自动感知应用程序属性,将相同属性应用程序的Container按照动态资源计划均衡分配到集群计算节点上,并使用NSGA-Ⅲ算法完成最优分配方案计算.实验结果表明,APB Scheduler解决了相同属性应用程序的Container分配倾斜问题,大幅提升了集群的性能和稳定性.
Abstract
Hadoop applications have computation intensive and scheduling time attributes.However,the three built-in resource schedulers of the second generation resource manager,which is integrated in Hadoop big data platform,are unable to evenly distribute applications with the same attributes to the computing nodes,resulting in excessive load on some nodes and serious long tail effect of computing tasks.This paper presents an application attribute aware yarn load balancing scheduling model-APB Scheduler.APB Schedu-ler automatically perceive the application attributes,evenly allocate the containers of the same attribute ap-plication to the cluster computing nodes according to the dynamic resource plan,and use the NSGA-Ⅲ al-gorithm to complete the calculation of the optimal allocation scheme.Through experimental verification,APB Scheduler solves the container allocation skew problem of applications with the same content,and greatly improves the performance and stability of the cluster.
关键词
NSGA-Ⅲ算法/Yarn/资源调度Key words
NSGA-Ⅲ/Yarn/resource scheduling引用本文复制引用
基金项目
陕西省自然科学基金(2020JM-637)
陕西省教育科学规划课题(十四五)(SGH21Y0303)
陕西省高等教育教学改革研究项目(21ZY015)
陕西省教育科学规划课题研究项目(十三五)(SGH20Y1420)
出版年
2024
NETL
NSTL
万方数据