English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We explore different ways to utilize position-based cross-attention inseq2seq networks to enable length generalization in algorithmic tasks. We showthat a simple approach of interpolating the original and reversed encodedrepresentations combined with relative attention allows near-perfect lengthgeneralization for both forward and reverse lookup tasks or copy tasks that hadbeen generally hard to tackle. We also devise harder diagnostic tasks where therelative distance of the ideal attention position varies with timestep. In suchsettings, the simple interpolation trick with relative attention is notsufficient. We introduce novel variants of location attention building on topof Dubois et al. (2020) to address the new diagnostic tasks. We also show thebenefits of our approaches for length generalization in SCAN (Lake & Baroni,2018) and CFQ (Keysers et al., 2020). Our code is available on GitHub.

Monotonic Location Attention for Length Generalization