English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Continual pre-training has been urgent for adapting a pre-trained model to amultitude of domains and tasks in the fast-evolving world. In practice, acontinually pre-trained model is expected to demonstrate not only greatercapacity when fine-tuned on pre-trained domains but also a non-decreasingperformance on unseen ones. In this work, we first investigate such anytimefine-tuning effectiveness of existing continual pre-training approaches,concluding with unanimously decreased performance on unseen domains. To thisend, we propose a prompt-guided continual pre-training method, where we train ahypernetwork to generate domain-specific prompts by both agreement anddisagreement losses. The agreement loss maximally preserves the generalizationof a pre-trained model to new domains, and the disagreement one guards theexclusiveness of the generated hidden states for each domain. Remarkably,prompts by the hypernetwork alleviate the domain identity when fine-tuning andpromote knowledge transfer across domains. Our method achieved improvements of3.57% and 3.4% on two real-world datasets (including domain shift and temporalshift), respectively, demonstrating its efficacy.

Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompt