English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While neural language models often perform surprisingly well on naturallanguage understanding (NLU) tasks, their strengths and limitations remainpoorly understood. Controlled synthetic tasks are thus an increasinglyimportant resource for diagnosing model behavior. In this work we focus onstory understanding, a core competency for NLU systems. However, the mainsynthetic resource for story understanding, the bAbI benchmark, lacks such asystematic mechanism for controllable task generation. We develop Dyna-bAbI, adynamic framework providing fine-grained control over task generation in bAbI.We demonstrate our ideas by constructing three new tasks requiringcompositional generalization, an important evaluation setting absent from theoriginal benchmark. We tested both special-purpose models developed for bAbI aswell as state-of-the-art pre-trained methods, and found that while bothapproaches solve the original tasks (>99% accuracy), neither approach succeededin the compositional generalization setting, indicating the limitations of theoriginal training data. We explored ways to augment the original data, andfound that though diversifying training data was far more useful than simplyincreasing dataset size, it was still insufficient for driving robustcompositional generalization (with <70% accuracy for complex compositions). Ourresults underscore the importance of highly controllable task generators forcreating robust NLU systems through a virtuous cycle of model and datadevelopment.

Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking