Premium
FireWorks: a dynamic workflow system designed for high‐throughput applications
Author(s) -
Jain Anubhav,
Ong Shyue Ping,
Chen Wei,
Medasani Bharat,
Qu Xiaohui,
Kocher Michael,
Brafman Miriam,
Petretto Guido,
Rignanese GianMarco,
Hautier Geoffroy,
Gunter Daniel,
Persson Kristin A.
Publication year - 2015
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.3505
Subject(s) - workflow , computer science , supercomputer , python (programming language) , throughput , fireworks , software , operating system , nosql , database , software engineering , big data , chemistry , organic chemistry , wireless
Summary This paper introduces FireWorks, a workflow software for running high‐throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU‐hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high‐throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long‐running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data‐driven and high‐throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work. Copyright © 2015 John Wiley & Sons, Ltd.