English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

To improve the explainability of leading Transformer networks used in NLP, itis important to tease apart genuine symbolic rules from merely associativeinput-output patterns. However, we identify several inconsistencies in how``symbolicity'' has been construed in recent NLP literature. To mitigate thisproblem, we propose two criteria to be the most relevant, one pertaining to asystem's internal architecture and the other to the dissociation betweenabstract rules and specific input identities. From this perspective, wecritically examine prior work on the symbolic capacities of Transformers, anddeem the results to be fundamentally inconclusive for reasons inherent inexperiment design. We further maintain that there is no simple fix to thisproblem, since it arises -- to an extent -- in all end-to-end settings.Nonetheless, we emphasize the need for more robust evaluation of whethernon-symbolic explanations exist for success in seemingly symbolic tasks. Tofacilitate this, we experiment on four sequence modelling tasks on the T5Transformer in two experiment settings: zero-shot generalization, andgeneralization across class-specific vocabularies flipped between the trainingand test set. We observe that T5's generalization is markedly stronger insequence-to-sequence tasks than in comparable classification tasks. Based onthis, we propose a thus far overlooked analysis, where the Transformer itselfdoes not need to be symbolic to be part of a symbolic architecture as theprocessor, operating on the input and output as external memory components.

Do Transformers know symbolic rules, and would we know if they did?