English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Modern big data frameworks (such as Hadoop and Spark) allow multiple users todo large-scale analysis simultaneously. Typically, users deploy Data-IntensiveWorkflows (DIWs) for their analytical tasks. These DIWs of different usersshare many common parts (i.e, 50-80%), which can be materialized to reuse themin future executions. The materialization improves the overall processing timeof DIWs and also saves computational resources. Current solutions formaterialization store data on Distributed File Systems (DFS) by using a fixeddata format. However, a fixed choice might not be the optimal one for everysituation. For example, it is well-known that different data fragmentationstrategies (i.e., horizontal, vertical or hybrid) behave better or worseaccording to the access patterns of the subsequent operations.  In this paper, we present a cost-based approach which helps deciding the mostappropriate storage format in every situation. A generic cost-based storageformat selector framework considering the three fragmentation strategies ispresented. Then, we use our framework to instantiate cost models for specificHadoop data formats (namely SequenceFile, Avro and Parquet), and test it withrealistic use cases. Our solution gives on average 33% speedup overSequenceFile, 11% speedup over Avro, 32% speedup over Parquet, and overall, itprovides upto 25% performance gain.

A Cost-based Storage Format Selector for Materialization in Big Data Frameworks