English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Data pipelines are widely employed in modern enterprises to power a varietyof Machine-Learning (ML) and Business-Intelligence (BI) applications.Crucially, these pipelines are \emph{recurring} (e.g., daily or hourly) inproduction settings to keep data updated so that ML models can be re-trainedregularly, and BI dashboards refreshed frequently. However, data quality (DQ)issues can often creep into recurring pipelines because of upstream schema anddata drift over time. As modern enterprises operate thousands of recurringpipelines, today data engineers have to spend substantial efforts to\emph{manually} monitor and resolve DQ issues, as part of their DataOps andMLOps practices.  Given the high human cost of managing large-scale pipeline operations, it isimperative that we can \emph{automate} as much as possible. In this work, wepropose Auto-Validate-by-History (AVH) that can automatically detect DQ issuesin recurring pipelines, leveraging rich statistics from historical executions.We formalize this as an optimization problem, and develop constant-factorapproximation algorithms with provable precision guarantees. Extensiveevaluations using 2000 production data pipelines at Microsoft demonstrate theeffectiveness and efficiency of AVH.

Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines