Open AccessEfficient and fair scheduler of multiple resources for MapReduce system
Author(s) -
Liao Jianxin,
Zhang Lei,
Li Tonghong,
Wang Jingyu,
Qi Qi
Publication year - 2016
Publication title -
iet software
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.305
H-Index - 43
eISSN - 1751-8814
pISSN - 1751-8806
DOI - 10.1049/iet-sen.2015.0109
Subject(s) - computer science , parallel computing , processor scheduling , distributed computing , scheduling (production processes) , computer architecture , resource (disambiguation) , computer network , operations management , engineering
Scheduling tasks close to their data and optimising resources utilisation are both crucial for the efficiency of MapReduce system. On the other hand, there is a conflict between fairness and efficiency. In this study, an efficient and dominant resource held time fairness (EHTF) scheduler is presented, in which the efficient utilisation of resources, data locality and fairness are addressed simultaneously. In EHTF scheduler, the authors introduce the concept of ‘coarse‐grained fairness’ to improve the efficiency of MapReduce system. For each scheduling, several tasks from different jobs can be assigned to the free slot without violating the coarse‐grained fairness doctrine. To determine the best task from these several tasks in each scheduling step, a score model is proposed by taking into consideration both resources utilisation and data locality. The authors describe the design and implementation of EHTF scheduler. The authors’ experimental results show that EHTF achieves more fairness and better throughput than Fair and Quincy schedulers.