English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Layer normalization (LN) is a widely adopted deep learning techniqueespecially in the era of foundation models. Recently, LN has been shown to besurprisingly effective in federated learning (FL) with non-i.i.d. data.However, exactly why and how it works remains mysterious. In this work, wereveal the profound connection between layer normalization and the label shiftproblem in federated learning. To understand layer normalization better in FL,we identify the key contributing mechanism of normalization methods in FL,called feature normalization (FN), which applies normalization to the latentfeature representation before the classifier head. Although LN and FN do notimprove expressive power, they control feature collapse and local overfittingto heavily skewed datasets, and thus accelerates global training. Empirically,we show that normalization leads to drastic improvements on standard benchmarksunder extreme label shift. Moreover, we conduct extensive ablation studies tounderstand the critical factors of layer normalization in FL. Our resultsverify that FN is an essential ingredient inside LN to significantly improvethe convergence of FL while remaining robust to learning rate choices,especially under extreme label shift where each client has access to fewclasses.

Normalization Is All You Need: Understanding Layer-Normalized Federated Learning under Extreme Label Shift