English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Prompt-based continual learning is an emerging direction in leveragingpre-trained knowledge for downstream continual learning, and has almost reachedthe performance pinnacle under supervised pre-training. However, our empiricalresearch reveals that the current strategies fall short of their full potentialunder the more realistic self-supervised pre-training, which is essential forhandling vast quantities of unlabeled data in practice. This is largely due tothe difficulty of task-specific knowledge being incorporated into instructedrepresentations via prompt parameters and predicted by uninstructedrepresentations at test time. To overcome the exposed sub-optimality, weconduct a theoretical analysis of the continual learning objective in thecontext of pre-training, and decompose it into hierarchical components:within-task prediction, task-identity inference, and task-adaptive prediction.Following these empirical and theoretical insights, we propose HierarchicalDecomposition (HiDe-)Prompt, an innovative approach that explicitly optimizesthe hierarchical components with an ensemble of task-specific prompts andstatistics of both uninstructed and instructed representations, further withthe coordination of a contrastive regularization strategy. Our extensiveexperiments demonstrate the superior performance of HiDe-Prompt and itsrobustness to pre-training paradigms in continual learning (e.g., up to 15.01%and 9.61% lead on Split CIFAR-100 and Split ImageNet-R, respectively). Our codeis available at \url{https://github.com/thu-ml/HiDe-Prompt}.

Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality