English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In the era of big data and cloud computing, large amounts of data aregenerated from user applications and need to be processed in the datacenter.Data-parallel computing frameworks, such as Apache Spark, are widely used toperform such data processing at scale. Specifically, Spark leveragesdistributed memory to cache the intermediate results, represented as ResilientDistributed Datasets (RDDs). This gives Spark an advantage over other parallelframeworks for implementations of iterative machine learning and data miningalgorithms, by avoiding repeated computation or hard disk accesses to retrieveRDDs. By default, caching decisions are left at the programmer's discretion,and the LRU policy is used for evicting RDDs when the cache is full. However,when the objective is to minimize total work, LRU is woefully inadequate,leading to arbitrarily suboptimal caching decisions. In this paper, we designan algorithm for multi-stage big data processing platforms to adaptivelydetermine and cache the most valuable intermediate datasets that can be reusedin the future. Our solution automates the decision of which RDDs to cache: thisamounts to identifying nodes in a direct acyclic graph (DAG) representingcomputations whose outputs should persist in the memory. Our experiment resultsshow that our proposed cache optimization solution can improve the performanceof machine learning applications on Spark decreasing the total work torecompute RDDs by 12%.

Intermediate Data Caching Optimization for Multi-Stage and Parallel Big Data Frameworks