English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

De-duplication---identification of distinct records referring to the samereal-world entity---is a well-known challenge in data integration. Since verylarge datasets prohibit the comparison of every pair of records, {\em blocking}has been identified as a technique of dividing the dataset for pairwisecomparisons, thereby trading off {\em recall} of identified duplicates for {\emefficiency}. Traditional de-duplication tasks, while challenging, typicallyinvolved a fixed schema such as Census data or medical records. However, withthe presence of large, diverse sets of structured data on the web and the needto organize it effectively on content portals, de-duplication systems need toscale in a new dimension to handle a large number of schemas, tasks and datasets, while handling ever larger problem sizes. In addition, when working in amap-reduce framework it is important that canopy formation be implemented as a{\em hash function}, making the canopy design problem more challenging. Wepresent CBLOCK, a system that addresses these challenges. CBLOCK learns hashfunctions automatically from attribute domains and a labeled dataset consistingof duplicates. Subsequently, CBLOCK expresses blocking functions using ahierarchical tree structure composed of atomic hash functions. The applicationmay guide the automated blocking process based on architectural constraints,such as by specifying a maximum size of each block (based on memoryrequirements), impose disjointness of blocks (in a grid environment), orspecify a particular objective function trading off recall for efficiency. As apost-processing step to automatically generated blocks, CBLOCK {\em rolls-up}smaller blocks to increase recall. We present experimental results on twolarge-scale de-duplication datasets at Yahoo!---consisting of over 140K moviesand 40K restaurants respectively---and demonstrate the utility of CBLOCK.

CBLOCK: An Automatic Blocking Mechanism for Large-Scale De-duplication Tasks