An Empirical Study of Hadoop's Energy Efficiency on a HPC Cluster | Zendy

Nidhi Tiwari | Zendy; Santonu Sarkar | Zendy; Umesh Bellur | Zendy; Maria Indrawan | Zendy

AI Assistant Blog Pricing

Home ZAIA Blog

Open Access

An Empirical Study of Hadoop's Energy Efficiency on a HPC Cluster

Author(s) -

Nidhi Tiwari,

Santonu Sarkar,

Umesh Bellur,

Maria Indrawan

Publication year - 2014

Publication title -

procedia computer science

Language(s) - English

Resource type - Journals

SCImago Journal Rank - 0.334

H-Index - 76

ISSN - 1877-0509

DOI - 10.1016/j.procs.2014.05.006

Subject(s) - computer science , efficient energy use , parallel computing , energy consumption , bandwidth (computing) , data parallelism , distributed computing , parallelism (grammar) , supercomputer , cluster (spacecraft) , computer network , ecology , electrical engineering , biology , engineering

Map-Reduce programming model is commonly used for efficient scientific computations, as it executes tasks in parallel and distributed manner on large data volumes. The HPC infrastructure can effectively increase the parallelism of map-reduce tasks. However such an execution will incur high energy and data transmission costs. Here we empirically study how the energy efficiency of a map-reduce job varies with increase in parallelism and network bandwidth on a HPC cluster. We also investigate the effectiveness of power-aware systems in managing the energy consumption of different types of map-reduce jobs. We comprehend that for some jobs the energy efficiency degrades at high degree of parallelism, and for some it improves at low CPU frequency. Consequently we suggest strategies for configuring the degree of parallelism, network bandwidth and power management features in a HPC cluster for energy efficient execution of map-reduce jobs

The content you want is available to Zendy users.

Already have an account? Click here to sign in.

Having issues? You can contact us here

Accelerating Research