Premium
Application health monitoring for extreme‐scale resiliency using cooperative fault management
Author(s) -
Agarwal Pratul K.,
Naughton Thomas,
Park Byung H.,
Bernholdt David E.,
Hursey Joshua J.,
Geist Al
Publication year - 2019
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.5449
Subject(s) - scalability , computer science , fault tolerance , backplane , distributed computing , supercomputer , scale (ratio) , fault (geology) , software , fault detection and isolation , reliability engineering , parallel computing , database , engineering , artificial intelligence , operating system , physics , quantum mechanics , seismology , actuator , geology
Summary Resiliency is and will be a critical factor in determining scientific productivity on current and exascale supercomputers, and beyond. Applications oblivious to and incapable of handling transient soft and hard errors could waste supercomputing resources or, worse, yield misleading scientific insights. We introduce a novel application‐driven silent error detection and recovery strategy based on application health monitoring. Our methodology uses application output that follows known patterns, as indicators of an application's health and knowledge that violation of these patterns could be indication of faults. Information from system monitors that report hardware and software health status is used to corroborate faults. Collectively, this information is used by a fault coordinator agent to take preventive and corrective measures by applying computational steering to an application between checkpoints. This cooperative fault management system uses the Fault Tolerance Backplane as a communication channel. The benefits of this framework are demonstrated with two real application case studies, molecular dynamics, and quantum chemistry simulations, on scalable clusters with simulated memory and I/O corruptions. The developed approach is general and can be easily applied to other applications.