English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recent studies on transfer learning have shown that selectively fine-tuning asubset of layers or customizing different learning rates for each layer cangreatly improve robustness to out-of-distribution (OOD) data and retaingeneralization capability in the pre-trained models. However, most of thesemethods employ manually crafted heuristics or expensive hyper-parametersearches, which prevent them from scaling up to large datasets and neuralnetworks. To solve this problem, we propose Trainable Projected Gradient Method(TPGM) to automatically learn the constraint imposed for each layer for afine-grained fine-tuning regularization. This is motivated by formulatingfine-tuning as a bi-level constrained optimization problem. Specifically, TPGMmaintains a set of projection radii, i.e., distance constraints between thefine-tuned model and the pre-trained model, for each layer, and enforces themthrough weight projections. To learn the constraints, we propose a bi-leveloptimization to automatically learn the best set of projection radii in anend-to-end manner. Theoretically, we show that the bi-level optimizationformulation could explain the regularization capability of TPGM. Empirically,with little hyper-parameter search cost, TPGM outperforms existing fine-tuningmethods in OOD performance while matching the best in-distribution (ID)performance. For example, when fine-tuned on DomainNet-Real and ImageNet,compared to vanilla fine-tuning, TPGM shows $22\%$ and $10\%$ relative OODimprovement respectively on their sketch counterparts. Code is available at\url{https://github.com/PotatoTian/TPGM}.

Trainable Projected Gradient Method for Robust Fine-tuning