Premium
A sliding window‐based dynamic load balancing for heterogeneous Hadoop clusters
Author(s) -
Liu Yang,
Jing Weizhe,
Liu Youbo,
Lv Lin,
Qi Man,
Xiang Yang
Publication year - 2016
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.3763
Subject(s) - computer science , load balancing (electrical power) , scalability , distributed computing , sliding window protocol , window (computing) , fault tolerance , cloud computing , cluster (spacecraft) , computer cluster , parallel computing , operating system , geometry , mathematics , grid
Summary At present MapReduce computing model‐based Hadoop framework has gradually become the most famous distributed computing framework because of its remarkable features such as scalability, fault tolerance, data security, and powerful IO ability. However, Hadoop framework only supports limited load balancing policies, which may result in performance deterioration in heterogeneous clusters. Additionally Hadoop does not have advanced dynamic load balancing mechanism in enabling its optimal performance in dynamic environment. This paper presents a sliding window‐based dynamic load balancing algorithm, which specially aims at balancing the load among the heterogeneous nodes during the Hadoop job processing. The presented algorithm is evaluated in both simulated and physical environments. The experimental results show that the performances in terms of efficiency of Hadoop cluster can be significantly improved. Copyright © 2016 John Wiley & Sons, Ltd.