English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large-scale vision-language (V-L) models have demonstrated remarkablegeneralization capabilities for downstream tasks through prompt tuning.However, the mechanisms behind the learned text representations are unknown,limiting further generalization gains, especially under class imbalancescenarios. Recent advances in the neural collapse (NC) phenomenon ofvision-only models suggest that the optimal representation structure is thesimplex ETF, which paves the way to study representations in V-L models. Inthis paper, we make the first attempt to use NC for examining therepresentations in V-L models via prompt tuning. It is found that NC optimalityof text-to-image representations shows a positive correlation with downstreamgeneralizability, which is more severe under class imbalance settings. Toimprove the representations, we propose Neural-collapse-anchored Prompt Tuning(NPT), a novel method that learns prompts with text and image representationsthat satisfy the same simplex ETF. NPT incorporates two regularization terms:language-modality collapse and multi-modality isomorphism; and it is compatiblewith other prompt tuning methods. Extensive experiments show that NPT canconsistently help to improve existing prompt tuning techniques across 11datasets for both balanced and imbalanced settings.

Understanding Prompt Tuning for V-L Models Through the Lens of Neural Collapse