首页|Courier: A Unified Communication Agent to Support Concurrent Flow Scheduling in Cluster Computing
Courier: A Unified Communication Agent to Support Concurrent Flow Scheduling in Cluster Computing
扫码查看
点击上方二维码区域，可以放大扫码查看
原文链接
NETL
NSTL
IEEE
As one of the pillars in cluster computing frameworks, coflow scheduling algorithms can effectively shorten the network transmission time of cluster computing jobs, thus reducing the job completion times and improving the execution performance. However, most of existing coflow scheduling algorithms failed to consider the influences of concurrent flows, which can degrade their performance under a massive number of concurrent flows. To fill the gap, we propose a unified communication agent named Courier to minimize the number of concurrent flows in cluster computing applications, which is compatible with the mainstream coflow scheduling approaches. To maintain the scheduling order given by the scheduling algorithms, Courier merges multiple flows between each pair of hosts into a unified flow, and determines its order based on that of origin flows. In addition, in order to adapt to various types of topologies, Courier introduces a control mechanism to adjust the number of flows while maintaining the scheduling order. Extensive large-scale trace-driven simulations have shown that Courier is compatible with existing scheduling algorithms, and outperforms the state-of-the-art approaches by about 30% under a variety of workloads and topologies.
Scheduling algorithmsCluster computingData centersTopologyServersSwitchesSchedulingTrainingNetwork topologyPacket loss
NETL
NSTL
IEEE
