English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformers have achieved remarkable success in several domains, rangingfrom natural language processing to computer vision. Nevertheless, it has beenrecently shown that stacking self-attention layers - the distinctivearchitectural component of Transformers - can result in rank collapse of thetokens' representations at initialization. The question of if and how rankcollapse affects training is still largely unanswered, and its investigation isnecessary for a more comprehensive understanding of this architecture. In thiswork, we shed new light on the causes and the effects of this phenomenon.First, we show that rank collapse of the tokens' representations hinderstraining by causing the gradients of the queries and keys to vanish atinitialization. Furthermore, we provide a thorough description of the origin ofrank collapse and discuss how to prevent it via an appropriate depth-dependentscaling of the residual branches. Finally, our analysis unveils that specificarchitectural hyperparameters affect the gradients of queries and valuesdifferently, leading to disproportionate gradient norms. This suggests anexplanation for the widespread use of adaptive methods for Transformers'optimization.

Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse