English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, efficient fine-tuning of large-scale pre-trained models hasattracted increasing research interests, where linear probing (LP) as afundamental module is involved in exploiting the final representations fortask-dependent classification. However, most of the existing methods focus onhow to effectively introduce a few of learnable parameters, and little workpays attention to the commonly used LP module. In this paper, we propose anovel Moment Probing (MP) method to further explore the potential of LP.Distinguished from LP which builds a linear classification head based on themean of final features (e.g., word tokens for ViT) or classification tokens,our MP performs a linear classifier on feature distribution, which provides thestronger representation ability by exploiting richer statistical informationinherent in features. Specifically, we represent feature distribution by itscharacteristic function, which is efficiently approximated by using first- andsecond-order moments of features. Furthermore, we propose a multi-headconvolutional cross-covariance (MHC$^3$) to compute second-order moments in anefficient and effective manner. By considering that MP could affect featurelearning, we introduce a partially shared module to learn two recalibratingparameters (PSRP) for backbones based on MP, namely MP$_{+}$. Extensiveexperiments on ten benchmarks using various models show that our MPsignificantly outperforms LP and is competitive with counterparts at lesstraining cost, while our MP$_{+}$ achieves state-of-the-art performance.

Tuning Pre-trained Model via Moment Probing