Open AccessDynamic Load Aware Scheduler of Map Reduce Tasks for Cloud Environments
Author(s) -
Adepu Sree Lakshmi,
N. Subhash Chandra,
M. BalRaju,
Adepu Sree Lakshmi,
Naveen Chandra
Publication year - 2019
Publication title -
international journal of recent technology and engineering
Language(s) - English
Resource type - Journals
ISSN - 2277-3878
DOI - 10.35940/ijrte.b1079.0982s1119
Subject(s) - cloud computing , computer science , virtualization , virtual machine , distributed computing , scheduling (production processes) , overhead (engineering) , big data , schedule , job scheduler , operating system , engineering , operations management
Most of the current day applications are data and compute intensive which led to invention of technologies like Hadoop. Hadoop uses Map Reduce framework for parallel processing of big data applications using the computing resources of multiple nodes. Hadoop is designed for cluster environments and has few limitations when executed in cloud environments. Hadoop on cloud has become a common choice due to its easy establishment of infrastructure and pay as you use model. Hadoop performance on cloud infrastructures is affected by the virtualization overhead of cloud environment. The execution times of Hadoop on cloud can be improved if the virtual resources are effectively used to schedule the tasks by studying the resource usage characteristics of the tasks and resource availability of the nodes. The proposed work is to build a dynamic scheduler for Hadoop framework which can make scheduling decision dynamically based on job resource usage and node load. The results of the proposed work indicate an improvement of up to 23% in execution time of the Hadoop Map Reduce applications.