English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Fine-tuning a pretrained transformer for a downstream task has become astandard method in NLP in the last few years. While the results from thesemodels are impressive, applying them can be extremely computationallyexpensive, as is pretraining new models with the latest architectures. Wepresent a novel method for applying pretrained transformer language modelswhich lowers their memory requirement both at training and inference time. Anadditional benefit is that our method removes the fixed context size constraintthat most transformer models have, allowing for more flexible use. When appliedto the GPT-2 language model, we find that our method attains better perplexitythan an unmodified GPT-2 model on the PG-19 and WikiText-103 corpora, for agiven amount of computation or memory.

Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size