English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This article derives and validates three principles for initialization andarchitecture selection in finite width graph neural networks (GNNs) with ReLUactivations. First, we theoretically derive what is essentially the uniquegeneralization to ReLU GNNs of the well-known He-initialization. Ourinitialization scheme guarantees that the average scale of network outputs andgradients remains order one at initialization. Second, we prove in finite widthvanilla ReLU GNNs that oversmoothing is unavoidable at large depth when usingfixed aggregation operator, regardless of initialization. We then prove thatusing residual aggregation operators, obtained by interpolating a fixedaggregation operator with the identity, provably alleviates oversmoothing atinitialization. Finally, we show that the common practice of using residualconnections with a fixup-type initialization provably avoids correlationcollapse in final layer features at initialization. Through ablation studies wefind that using the correct initialization, residual aggregation operators, andresidual connections in the forward pass significantly and reliably speeds upearly training dynamics in deep ReLU GNNs on a variety of tasks.

Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations