Premium
A Comprehensive Python Toolkit for Accessing High‐Throughput Computing to Support Large Hydrologic Modeling Tasks
Author(s) -
Christensen Scott D.,
Swain Nathan R.,
Jones Norman L.,
Nelson E. James,
Snow Alan D.,
Dolder Herman G.
Publication year - 2017
Publication title -
jawra journal of the american water resources association
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.957
H-Index - 105
eISSN - 1752-1688
pISSN - 1093-474X
DOI - 10.1111/1752-1688.12455
Subject(s) - python (programming language) , computer science , cloud computing , provisioning , interoperability , scalability , hydrological modelling , software engineering , database , world wide web , operating system , climatology , geology
The National Flood Interoperability Experiment (NFIE) was an undertaking that initiated a transformation in national hydrologic forecasting by providing streamflow forecasts at high spatial resolution over the whole country. This type of large‐scale, high‐resolution hydrologic modeling requires flexible and scalable tools to handle the resulting computational loads. While high‐throughput computing (HTC) and cloud computing provide an ideal resource for large‐scale modeling because they are cost‐effective and highly scalable, nevertheless, using these tools requires specialized training that is not always common for hydrologists and engineers. In an effort to facilitate the use of HTC resources the National Science Foundation (NSF) funded project, CI‐WATER, has developed a set of Python tools that can automate the tasks of provisioning and configuring an HTC environment in the cloud, and creating and submitting jobs to that environment. These tools are packaged into two Python libraries: CondorPy and TethysCluster. Together these libraries provide a comprehensive toolkit for accessing HTC to support hydrologic modeling. Two use cases are described to demonstrate the use of the toolkit, including a web app that was used to support the NFIE national‐scale modeling.