Research and application of big data platform technology based on multi-centre collaborative computing
China Telecom has launched a high-efficient and collaborative wide-area big data architecture system,the cloud edge computing big data platform,for large-scale governmental and enterprise organizations spanning multiple geographies and clusters.The platform logically abstracts data partitions through the cluster dimension,integrates multiple independent datasets into a"virtual dataset",and achieves many-to-one dataset mapping management.At the same time,the computing load dataset of the platform has generalized characteristics,which can flexibly cope with the data processing requirements in different scenarios.In addition,the platform also supports a variety of computing engines and scheduling systems using relational expressions as intermediate representations to achieve batch tasks for large-scale,complex data processing in highly fault-tolerant scenarios.At present,the cloud edge computing big data platform has been applied in a variety of application scenes.The platform has improved efficiency by 17%in 5G Core capacity scheduling subsystem(5GC)multi-centre big data job development and operation,and has achieved the col-laborative scheduling of a total of 42 PB of storage,84 TB of memory,and 24 984 VCore computing resources,with a daily average of 80 308 times of task scheduling between the front cluster and the core cluster.
cloud edge collaborationuniform SQLtask optimizationbig data platform
万方数据
维普
