We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
基于禁忌搜索的流式计算平台负载均衡.
- Authors
王英杰; 李梓杨; 于炯; 陈鹏程
- Abstract
Focused on the problem of unbalanced computing load distribution and low resource utilization in the native scheduling mechanism of big data streaming computing platform, a load balancing strategy based on Tabu search algorithm in heterogeneous environments is proposed and applied to the Apache Flink platform. Firstly, this strategy sets up a job topology model and abstracts the topology of streaming computing jobs as a directed acyclic graph.Therefore, each task slot becomes a node, which lays the foundation for performance evaluation of computing nodes.Secondly, the method imports the performance evaluation model to nodes with performance weights in the directed acyclic graph, and obtains the performance of the nodes through normalization processing; then the evaluation parameters are passed into the Tabu Search for job path optimization, so as to obtain the optimal job path. Finally, by using the CustomizationWrapper interface, this strategy allocates data to the nodes included in the optimal job path and completes the balancing of computational load. The algorithm then passes evaluation parameters into the tabu scheduling algorithm for job path optimization, thereby obtaining the optimal job path. The experimental results show that the load balancing strategy optimized by the Tabu scheduling algorithm reduces the average computing latency by 10-20ms compared to the native Flink platform. The strategy significantly improves resource utilization, and increases average throughput by about 15%. This effectively proves the effectiveness and optimization effect of the load balancing strategy.
- Publication
Application Research of Computers / Jisuanji Yingyong Yanjiu, 2023, Vol 40, Issue 12, p3701
- ISSN
1001-3695
- Publication type
Article
- DOI
10.19734/j.issn.1001-3695.2023.04.0180