English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we explore the structure of the penultimate Gram matrix indeep neural networks, which contains the pairwise inner products of outputscorresponding to a batch of inputs. In several architectures it has beenobserved that this Gram matrix becomes degenerate with depth at initialization,which dramatically slows training. Normalization layers, such as batch or layernormalization, play a pivotal role in preventing the rank collapse issue.Despite promising advances, the existing theoretical results do not extend tolayer normalization, which is widely used in transformers, and can notquantitatively characterize the role of non-linear activations. To bridge thisgap, we prove that layer normalization, in conjunction with activation layers,biases the Gram matrix of a multilayer perceptron towards the identity matrixat an exponential rate with depth at initialization. We quantify this rateusing the Hermite expansion of the activation function.

On the impact of activation and normalization in obtaining isometric embeddings at initialization