English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Systematic compositionality is an essential mechanism in human language,allowing the recombination of known parts to create novel expressions. However,existing neural models have been shown to lack this basic ability in learningsymbolic structures. Motivated by the failure of a Transformer model on theSCAN compositionality challenge (Lake and Baroni, 2018), which requires parsinga command into actions, we propose two auxiliary sequence prediction tasks thattrack the progress of function and argument semantics, as additional trainingsupervision. These automatically-generated sequences are more representative ofthe underlying compositional symbolic structures of the input data. Duringinference, the model jointly predicts the next action and the next tokens inthe auxiliary sequences at each step. Experiments on the SCAN dataset show thatour method encourages the Transformer to understand compositional structures ofthe command, improving its accuracy on multiple challenging splits from <= 10%to 100%. With only 418 (5%) training instances, our approach still achieves97.8% accuracy on the MCD1 split. Therefore, we argue that compositionality canbe induced in Transformers given minimal but proper guidance. We also show thata better result is achieved using less contextualized vectors as theattention's query, providing insights into architecture choices in achievingsystematic compositionality. Finally, we show positive generalization resultson the groundedSCAN task (Ruis et al., 2020). Our code is publicly availableat: https://github.com/jiangycTarheel/compositional-auxseq

Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks