Premium
A distributed load balancing algorithm for climate big data processing over a multi‐core CPU cluster
Author(s) -
Wang Yuzhu,
Jiang Jinrong,
Ye Huang,
He Juanxiong
Publication year - 2016
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.3822
Subject(s) - weather research and forecasting model , speedup , computer science , computation , grid , load balancing (electrical power) , parallel computing , computational science , multi core processor , central processing unit , supercomputer , process (computing) , computer cluster , cluster (spacecraft) , real time computing , algorithm , distributed computing , meteorology , computer hardware , mathematics , programming language , physics , geometry , operating system
Summary Load imbalance is a common problem to be tackled urgently in large scale data‐driven simulation systems or data intensive computing. According to the coupler, the Chinese Academy of Sciences‐Earth System Model (CAS‐ESM) implements one‐way nesting of the Institute of Atmospheric Physics of Chinese Academy of Sciences Atmospheric General Circulation Model version 4.0 (IAP AGCM4.0) and Weather Research and Forecasting model (WRF). The METGRID (meteorological grid) and REAL program modules in the WRF are used to process meteorological data. In the CAS‐ESM, the load of the METGRID module is seriously unbalanced on many CPU cores. The load imbalance has a serious impact on the processing speed of meteorological data, so this study designs an optimization algorithm to solve the problem. Numerical experiments show that compared to before optimization, the optimization algorithm can solve the load imbalance of the METGRID, and the computation speed of the METGRID and REAL modules after optimization on 64 CPU cores is about 7.2 times faster than before. Meanwhile, the whole computation speed of the CAS‐ESM can improve by 217.53%. In addition, results indicate that they also can reach to a similar speedup on different numbers of CPU cores. Copyright © 2016 John Wiley & Sons, Ltd.