大数据2024,Vol.10Issue(2) :94-108.DOI:10.11959/j.issn.2096-0271.2023068

面向算力网络的跨域数据管理方法

Cross-domain data management for computing power networks

鲁蔚征 戴奇志 张策
大数据2024,Vol.10Issue(2) :94-108.DOI:10.11959/j.issn.2096-0271.2023068

面向算力网络的跨域数据管理方法

Cross-domain data management for computing power networks

鲁蔚征 1戴奇志 2张策3
扫码查看

作者信息

  • 1. 中国人民大学大型科学仪器共享平台,北京 100083
  • 2. 联旌智能科技(上海)有限公司,上海 200051
  • 3. 华中科技大学网络与计算中心,湖北 武汉 430074
  • 折叠

摘要

跨域算力网络希望整合多个算力中心的计算和数据资源,但现有的方案对跨域文件和数据管理关注不够.提出了一种轻量级的跨域算力网络数据管理方案:通过文件系统协议转换,接入远程算力中心的并行文件系统存储资源;算力中心内部的存储资源作为一种补充,应对高IOPS应用;通过容器绑定技术,将远程存储挂载并绑定到指定目录.基于该方案的原型系统已经在高校校级计算平台部署运行.实测数据和用户体验显示,该方案能够满足常见高性能计算应用需求.

Abstract

Cross-domain computing power networks wish to integrate computational and data resources from multiple computing centers,but existing methods do not pay enough attention to cross-domain file and data management.In this paper,a lightweight data access scheme for cross-domain computing power networks was proposed:(1)accessing parallel file system storage resources of remote computing centers through file system protocol conversion;(2)local caching as a supplement to cope with high IOPS applications;and(3)mounting remote or local storage to specified directories through container binding technology.The prototype system based on this scheme had been deployed on high-performance computing centers in multiple universities.The measured data and user experience showed that the scheme in this paper could meet the requirements of common high-performance computing applications.

关键词

算力网络/并行文件系统/数据管理/异构存储资源

Key words

computing power network/parallel file system/data management/heterogeneous storage resource

引用本文复制引用

基金项目

国家重点研发计划(2020YFB1710004)

出版年

2024
大数据
人民邮电出版社

大数据

CSTPCD
ISSN:2096-0271
参考文献量22
段落导航相关论文