面向负载均衡的动态均衡分区策略
DYNAMIC BALANCING PARTITION STRATEGY FOR LOAD BALANCING
杨迪 1赵家伟 1王鹏 1赵建平1
作者信息
摘要
针对MapReduce计算框架处理倾斜数据集时造成Reduce端出现负载不均衡现象,提出一种动态均衡分区策略.在mapper阶段提出基于分治法的数据切分原则处理任务传入的数据组;结合最佳适应算法思想设计动态分配原则逐步将切分后的数据块均衡分配到预分区链表中;根据分区索引分配到各Reduce节点上实现负载均衡.实验结果显示,动态均衡分区策略与两个基准模型相比任务执行时长平均降低了 7.7％,表明动态均衡分区策略更好地解决了数据倾斜问题,降低了任务执行时间,验证了模型的有效性.
Abstract
In view of the unbalanced load on the reduce side caused by MapReduce computing framework processing inclined data sets,this paper proposes a dynamic balanced partition strategy.In the mapper stage,the data segmentation principle based on divide and conquer method was proposed to process the incoming data groups.The dynamic allocation principle was designed combined with the idea of the best adaptive algorithm to gradually allocate the segmented data blocks to the server.According to the partition index,it was allocated to each reduce node to achieve load balancing.The experimental results show that,compared with the two benchmark models,the average task execution time of the dynamic balanced partition strategy is reduced by 7.7％,which indicates that the dynamic balanced partition strategy can better solve the problem of data skew,reduce the task execution time,and verify the effectiveness of the model.
关键词
MapReduce/负载均衡/数据倾斜/数据分区Key words
MapReduce/Load balancing/Data skew/Data partition引用本文复制引用
基金项目
中央引导地方科技发展资金吉林省基础研究专项(202002038JC)
出版年
2024
国家科技期刊平台
NSTL
万方数据