English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

State-of-the-art distributed stream processing systems such as Apache Flinkand Storm have recently included checkpointing to provide fault-tolerance forstateful applications. This is a necessary eventuality as these systems headinto the Exascale regime, and is evidently more efficient than replication asstate size grows. However current systems use a nominal value for thecheckpoint interval, indicative of assuming roughly 1 failure every 19 days,that does not take into account the salient aspects of the checkpoint process,nor the system scale, which can readily lead to inefficient system operation.To address this shortcoming, we provide a rigorous derivation of utilization --the fraction of total time available for the system to do useful work -- thatincorporates checkpoint interval, failure rate, checkpoint cost, failuredetection and restart cost, depth of the system topology and message delay. Ourmodel yields an elegant expression for utilization and provides an optimalcheckpoint interval given these parameters, interestingly showing it to bedependent only on checkpoint cost and failure rate. We confirm the accuracy andefficacy of our model through experiments with Apache Flink, where we obtainimprovements in system utilization for every case, especially as the systemsize increases. Our model provides a solid theoretical basis for the analysisand optimization of more elaborate checkpointing approaches.

A Utilization Model for Optimization of Checkpoint Intervals in Distributed Stream Processing Systems