English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Neural networks are often trained on multiple tasks, either simultaneously(multi-task learning, MTL) or sequentially (pretraining and subsequentfinetuning, PT+FT). In particular, it is common practice to pretrain neuralnetworks on a large auxiliary task before finetuning on a downstream task withfewer samples. Despite the prevalence of this approach, the inductive biasesthat arise from learning multiple tasks are poorly characterized. In this work,we address this gap. We describe novel implicit regularization penaltiesassociated with MTL and PT+FT in diagonal linear networks andsingle-hidden-layer ReLU networks. These penalties indicate that MTL and PT+FTinduce the network to reuse features in different ways. 1) Both MTL and PT+FTexhibit biases towards feature reuse between tasks, and towards sparsity in theset of learned features. We show a "conservation law" that implies a directtradeoff between these two biases. 2) PT+FT exhibits a novel "nested featureselection" regime, not described by either the "lazy" or "rich" regimesidentified in prior work, which biases it to rely on a sparse subset of thefeatures learned during pretraining. This regime is much narrower for MTL. 3)PT+FT (but not MTL) in ReLU networks benefits from features that are correlatedbetween the auxiliary and main task. We confirm these findings empirically withteacher-student models, and introduce a technique -- weight rescaling followingpretraining -- that can elicit the nested feature selection regime. Finally, wevalidate our theory in deep neural networks trained on image classification. Wefind that weight rescaling improves performance when it causes models todisplay signatures of nested feature selection. Our results suggest that nestedfeature selection may be an important inductive bias for finetuning neuralnetworks.

Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse