通信学报2024,Vol.45Issue(2) :188-200.DOI:10.11959/j.issn.1000-436x.2024006

流式大数据平台下的弹性数据迁移能效优化策略

Energy-efficient optimization strategy based on elastic data migration in big data streaming platform

蒲勇霖 许小龙 于炯 李梓杨 国冰磊
通信学报2024,Vol.45Issue(2) :188-200.DOI:10.11959/j.issn.1000-436x.2024006

流式大数据平台下的弹性数据迁移能效优化策略

Energy-efficient optimization strategy based on elastic data migration in big data streaming platform

蒲勇霖 1许小龙 1于炯 2李梓杨 2国冰磊3
扫码查看

作者信息

  • 1. 南京信息工程大学软件学院,江苏 南京 210044
  • 2. 新疆大学软件学院,新疆 乌鲁木齐 830002
  • 3. 湖北文理学院计算机工程学院,湖北 襄阳 441053
  • 折叠

摘要

针对流式计算框架在最初设计时缺乏能效方面的考虑,导致其存在高能耗与低效率的问题,提出一种流式大数据平台下的弹性数据迁移节能优化策略.首先,建立负载预测模型与资源判定模型,并进一步设计负载预测算法,通过预测负载变化趋势确定节点资源占用,找到资源过载与过剩节点;其次,建立资源约束模型与最优数据迁移模型,由此提出最优数据迁移算法,以提高节点资源利用率为目的进行数据迁移;最后,建立能耗模型,计算集群进行数据迁移后节约的能耗.实验结果表明,数据迁移节能优化策略能够对集群内节点资源变化做出及时响应,并在提高节点资源利用率的基础上,有效提高集群数据处理的能效.

Abstract

Focused on the problem that the stream computing platform was suffering from the high energy consumption and low efficiency due to the lack of consideration for energy efficiency in designing process,an energy-efficient optimi-zation strategy based on elastic data migration in big data streaming platform(EEDM-BDSP)was proposed.Firstly,models of the load prediction and the resource judgment were set up,and the load prediction algorithm was designed,which predicted the load tendency and determine node resource occupancy,so as to find nodes of resource overload and redundancy.Secondly,models of the resource constraint and the optimal data migration were set up,and the optimal data migration algorithm was proposed,which data migration for the purpose of improving node resource utilization.Finally,model of the energy consumption was set up to calculate the energy consumption saved by the cluster after data migration.The experimental results show that the EEDM-BDSP changes node resources in the cluster can responded on time,the resource utilization and the energy-efficient are improved.

关键词

流式计算/负载预测/资源约束/数据迁移/能效

Key words

stream computing/load prediction/resource constraint/data migration/energy-efficient

引用本文复制引用

基金项目

国家自然科学基金资助项目(62262064)

新疆维吾尔自治区重点研发计划基金资助项目(2022295358)

江苏省高等学校自然科学基金资助项目(23KJB520019)

新疆维吾尔自治区自然科学基金资助项目(2022D01C56)

湖北省自然科学基金资助项目(2022CFB805)

出版年

2024
通信学报
中国通信学会

通信学报

CSTPCD北大核心
影响因子:1.265
ISSN:1000-436X
参考文献量24
段落导航相关论文