English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Understanding the performance of data-parallel workloads whenresource-constrained has significant practical importance but unfortunately hasreceived only limited attention. This paper identifies, quantifies anddemonstrates memory elasticity, an intrinsic property of data-parallel tasks.Memory elasticity allows tasks to run with significantly less memory that theywould ideally want while only paying a moderate performance penalty. Forexample, we find that given as little as 10% of ideal memory, PageRank andNutchIndexing Hadoop reducers become only 1.2x/1.75x and 1.08x slower. We showthat memory elasticity is prevalent in the Hadoop, Spark, Tez and Flinkframeworks. We also show that memory elasticity is predictable in nature bybuilding simple models for Hadoop and extending them to Tez and Spark.  To demonstrate the potential benefits of leveraging memory elasticity, thispaper further explores its application to cluster scheduling. In this setting,we observe that the resource vs. time trade-off enabled by memory elasticitybecomes a task queuing time vs task runtime trade-off. Tasks may completefaster when scheduled with less memory because their waiting time is reduced.We show that a scheduler can turn this task-level trade-off into improved jobcompletion time and cluster-wide memory utilization. We have integrated memoryelasticity into Apache YARN. We show gains of up to 60% in average jobcompletion time on a 50-node Hadoop cluster. Extensive simulations show similarimprovements over a large number of scenarios.

Don't cry over spilled records: Memory elasticity of data-parallel applications and its application to cluster scheduling