English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently introduced language model prompting methods can achieve highaccuracy in zero- and few-shot settings while requiring few to no learnedtask-specific parameters. Nevertheless, these methods still often trail behindfull model finetuning. In this work, we investigate if a dedicated continuedpretraining stage could improve "promptability", i.e., zero-shot performancewith natural language prompts or few-shot performance with prompt tuning. Wereveal settings where existing continued pretraining methods lackpromptability. We also identify current methodological gaps, which we fill withthorough large-scale experiments. We demonstrate that a simple recipe,continued pretraining that incorporates a trainable prompt during multi-tasklearning, leads to improved promptability in both zero- and few-shot settingscompared to existing methods, up to 31% relative. On the other hand, we findthat continued pretraining using MAML-style meta-learning, a method thatdirectly optimizes few-shot promptability, yields subpar performance. Wevalidate our findings with two prompt tuning methods, and, based on ourresults, we provide concrete recommendations to optimize promptability fordifferent use cases.

Continued Pretraining for Better Zero- and Few-Shot Promptability