English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Enterprise data lakes often suffer from substantial amounts of duplicate andredundant data, with data volumes ranging from terabytes to petabytes. Thisleads to both increased storage costs and unnecessarily high maintenance costsfor these datasets. In this work, we focus on identifying and reducingredundancy in enterprise data lakes by addressing the problem of 'datasetcontainment'. To the best of our knowledge, this is one of the first works thataddresses table-level containment at a large scale.  We propose R2D2: a three-step hierarchical pipeline that efficientlyidentifies almost all instances of containment by progressively reducing thesearch space in the data lake. It first builds (i) a schema containment graph,followed by (ii) statistical min-max pruning, and finally, (iii) content levelpruning. We further propose minimizing the total storage and access costs byoptimally identifying redundant datasets that can be deleted (and reconstructedon demand) while respecting latency constraints.  We implement our system on Azure Databricks clusters using Apache Spark forenterprise data stored in ADLS Gen2, and on AWS clusters for open-source data.In contrast to existing modified baselines that are inaccurate or take severaldays to run, our pipeline can process an enterprise customer data lake at theTB scale in approximately 5 hours with high accuracy. We present theoreticalresults as well as extensive empirical validation on both enterprise (scale ofTBs) and open-source datasets (scale of MBs - GBs), which showcase theeffectiveness of our pipeline.

R2D2: Reducing Redundancy and Duplication in Data Lakes