English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Self-supervised pre-training of large-scale transformer models on textcorpora followed by finetuning has achieved state-of-the-art on a number ofnatural language processing tasks. Recently, Lu et al. (2021, arXiv:2103.05247)claimed that frozen pretrained transformers (FPTs) match or outperform trainingfrom scratch as well as unfrozen (fine-tuned) pretrained transformers in a setof transfer tasks to other modalities. In our work, we find that this resultis, in fact, an artifact of not tuning the learning rates. After carefullyredesigning the empirical setup, we find that when tuning learning ratesproperly, pretrained transformers do outperform or match training from scratchin all of our tasks, but only as long as the entire model is finetuned. Thus,while transfer from pretrained language models to other modalities does indeedprovide gains and hints at exciting possibilities for future work, properlytuning hyperparameters is important for arriving at robust findings.

Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers