English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Advances in the expressivity of pretrained models have increased interest inthe design of adaptation protocols which enable safe and effective transferlearning. Going beyond conventional linear probing (LP) and fine tuning (FT)strategies, protocols that can effectively control feature distortion, i.e.,the failure to update features orthogonal to the in-distribution, have beenfound to achieve improved out-of-distribution generalization (OOD). In order tolimit this distortion, the LP+FT protocol, which first learns a linear probeand then uses this initialization for subsequent FT, was proposed. However, inthis paper, we find when adaptation protocols (LP, FT, LP+FT) are alsoevaluated on a variety of safety objectives (e.g., calibration, robustness,etc.), a complementary perspective to feature distortion is helpful to explainprotocol behavior. To this end, we study the susceptibility of protocols tosimplicity bias (SB), i.e. the well-known propensity of deep neural networks torely upon simple features, as SB has recently been shown to underlie severalproblems in robust generalization. Using a synthetic dataset, we demonstratethe susceptibility of existing protocols to SB. Given the strong effectivenessof LP+FT, we then propose modified linear probes that help mitigate SB, andlead to better initializations for subsequent FT. We verify the effectivenessof the proposed LP+FT variants for decreasing SB in a controlled setting, andtheir ability to improve OOD generalization and safety on three adaptationdatasets.

A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias