English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider fine-tuning a pretrained deep neural network on a target task. Westudy the generalization properties of fine-tuning to understand the problem ofoverfitting, which has often been observed (e.g., when the target dataset issmall or when the training labels are noisy). Existing generalization measuresfor deep networks depend on notions such as distance from the initialization(i.e., the pretrained network) of the fine-tuned model and noise stabilityproperties of deep networks. This paper identifies a Hessian-based distancemeasure through PAC-Bayesian analysis, which is shown to correlate well withobserved generalization gaps of fine-tuned models. Theoretically, we proveHessian distance-based generalization bounds for fine-tuned models. We alsodescribe an extended study of fine-tuning against label noise, whereoverfitting remains a critical problem. We present an algorithm and ageneralization error guarantee for this algorithm under a class conditionalindependent noise model. Empirically, we observe that the Hessian-baseddistance measure can match the scale of the observed generalization gap offine-tuned models in practice. We also test our algorithm on several imageclassification tasks with noisy training labels, showing gains over priormethods and decreases in the Hessian distance measure of the fine-tuned model.

Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees