English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformers have achieved great success in machine learning applications.Normalization techniques, such as Layer Normalization (LayerNorm, LN) and RootMean Square Normalization (RMSNorm), play a critical role in accelerating andstabilizing the training of Transformers. While LayerNorm recenters andrescales input vectors, RMSNorm only rescales the vectors by their RMS value.Despite being more computationally efficient, RMSNorm may compromise therepresentation ability of Transformers. There is currently no consensusregarding the preferred normalization technique, as some models employLayerNorm while others utilize RMSNorm, especially in recent large languagemodels. It is challenging to convert Transformers with one normalization to theother type. While there is an ongoing disagreement between the twonormalization types, we propose a solution to unify two mainstream Transformerarchitectures, Pre-LN and Pre-RMSNorm Transformers. By removing the inherentredundant mean information in the main branch of Pre-LN Transformers, we canreduce LayerNorm to RMSNorm, achieving higher efficiency. We further proposethe Compressed RMSNorm (CRMSNorm) and Pre-CRMSNorm Transformer based on alossless compression of the zero-mean vectors. We formally establish theequivalence of Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm Transformer variants inboth training and inference. It implies that Pre-LN Transformers can besubstituted with Pre-(C)RMSNorm counterparts at almost no cost, offering thesame arithmetic functionality along with free efficiency improvement.Experiments demonstrate that we can reduce the training and inference time ofPre-LN Transformers by 1% - 10%.

Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers