We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
A frequency-aware and energy-saving strategy based on DVFS for Spark.
- Authors
Li, Hongjian; Wei, Yaojun; Xiong, Yu; Ma, Enjie; Tian, Wenhong
- Abstract
With the fast growth of big data applications, it has brought about a huge increase in the energy consumption for big data processing in Cloud data centers. In this study, a frequency-aware and energy-saving strategy based on dynamic voltage and frequency scaling (abbreviated as FAESS-DVFS) is proposed to reduce energy consumption for big data processing in Spark on YARN. Energy saving in two layers (YARN layer and Spark layer) has been designed and implemented for the proposed method. First, an optimal CPU frequency is presented in YARN layer based on the minimum energy efficiency ratio (EER) which can be obtained from status monitoring module. Then, a task scheduling method in Spark layer is constructed to optimize the energy consumption by dynamically adjusting the CPU frequency of nodes in the life cycle of different stages. Test on Hibench, the proposed method can achieve substantial energy saving of up to 29.5% for big data processing compared with the default algorithm in Spark on YARN while satisfying SLA constrains.
- Subjects
DATA processing service centers; ENERGY consumption; BIG data; ELECTRONIC data processing; ALGORITHMS; CONSUMPTION (Economics); SPARK ignition engines
- Publication
Journal of Supercomputing, 2021, Vol 77, Issue 10, p11575
- ISSN
0920-8542
- Publication type
Article
- DOI
10.1007/s11227-021-03740-5