Research on New Optimization Algorithm for Heterogeneous Hadoop Task Scheduling Based on Spider Monkey
In response to the problems of uneven task allocation and low resource utilization in current Hadoop task planning and scheduling,and considering the performance differences between nodes in a Hadoop cluster,this paper proposes a heterogeneous Hadoop scheduling algorithm based on spider monkey optimization.Firstly,in order to obtain the task load information of each node,this article adopts a cluster heartbeat mechanism to sequentially obtain the memory and CPU information of each node.Then,the spider monkey optimization algorithm mechanism is used to construct an objective function related to task completion time,and the optimal mapping relationship between task vol-ume and resource allocation is found.Finally,based on the task type and the resource utilization rate of nodes in the current cluster,combined with the optimal mapping relationship,the allocation and execution of new tasks are comple-ted.The experimental results show that compared with existing scheduling algorithms,the method proposed in this pa-per can effectively reduce task execution time,and improve scheduling efficiency,and task execution speed.
HeterogeneousTask schedulingSpider monkeyOptimization algorithmObjective function
NETL
NSTL
万方数据
