Stream Processing Task Scheduling Integrating Deep Reinforcement Learning and Operator Optimization
Aiming at the problems of high real-time requirements and large workload during the execution of stream processing jobs,the same processing fragments among multiple jobs might lead to repeated cal-culations of stream processing engines,waste of resources,and low processing performance,a scheduling method for stream processing tasks that integrated deep reinforcement learning and operator optimization was proposed.Firstly,the operator optimization algorithm was used to deduplicate and reconstruct multi-ple complex tasks.Secondly,the reconstructed tasks were input into a recurrent neural network to obtain the scheduling strategy for the tasks.Finally,the scheduling strategy was further optimized using a rein-forcement learning model.The proposed method reduced the number of operator instances created in each task through operator optimization.By combining deep reinforcement learning,the method automatically discovered the optimal scheduling strategy,effectively avoiding issues such as insufficient system re-sources and data congestion caused by a large number of instances running.The comparative experimen-tal results showed that the proposed method performed better in terms of throughput and latency.
万方数据
维普
