arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Computer Networks and Communications

Efficient Coflow Scheduling with Deep Reinforcement Learning and DAG Dependency Modeling

Yuanhao He, Xiaofeng Cheng, Yuan Liang, Xiaoyu Yi, Linlin Yan, Geyang Xiao

See on the map

Abstract

Cluster computing is widely used in fields such as massive data processing, data mining, and machine learning, it includes the computation stage and the communication stage. Cluster computing jobs include some computation stages and communication stages, typically modeled as DAGs. The communication stage between computing nodes transferring massive amounts of data has a significant impact on the time consumption of the whole job. Current research mostly optimize the completion time of the entire job by minimizing CCT or deadline missing rate without considering the time consumption of computation stages. Additionally, existing work neglect the expression of dependency relationship in the DAG. In this paper, in order to minimize JCT, we propose a reinforcement learning framework to generate coflow scheduling policies, taking into account the time consumption of computation stages. In addition, we introduce RNN to model the dependency between coflows and MPNN to generate the information representation of them. Finally, we achieve lower job completion time than FIFO in the real network environment, which proves that the coflow scheduling scheme proposed in this paper is more efficient.

More like this