English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Few-shot fine-tuning and in-context learning are two alternative strategiesfor task adaptation of pre-trained language models. Recently, in-contextlearning has gained popularity over fine-tuning due to its simplicity andimproved out-of-domain generalization, and because extensive evidence showsthat fine-tuned models pick up on spurious correlations. Unfortunately,previous comparisons of the two approaches were done using models of differentsizes. This raises the question of whether the observed weaker out-of-domaingeneralization of fine-tuned models is an inherent property of fine-tuning or alimitation of the experimental setup. In this paper, we compare thegeneralization of few-shot fine-tuning and in-context learning to challengedatasets, while controlling for the models used, the number of examples, andthe number of parameters, ranging from 125M to 30B. Our results show thatfine-tuned language models can in fact generalize well out-of-domain. We findthat both approaches generalize similarly; they exhibit large variation anddepend on properties such as model size and the number of examples,highlighting that robust task adaptation remains a challenge.

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation