English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The implicit bias towards solutions with favorable properties is believed tobe a key reason why neural networks trained by gradient-based optimization cangeneralize well. While the implicit bias of gradient flow has been widelystudied for homogeneous neural networks (including ReLU and leaky ReLUnetworks), the implicit bias of gradient descent is currently only understoodfor smooth neural networks. Therefore, implicit bias in non-smooth neuralnetworks trained by gradient descent remains an open question. In this paper,we aim to answer this question by studying the implicit bias of gradientdescent for training two-layer fully connected (leaky) ReLU neural networks. Weshowed that when the training data are nearly-orthogonal, for leaky ReLUactivation function, gradient descent will find a network with a stable rankthat converges to $1$, whereas for ReLU activation function, gradient descentwill find a neural network with a stable rank that is upper bounded by aconstant. Additionally, we show that gradient descent will find a neuralnetwork such that all the training data points have the same normalized marginasymptotically. Experiments on both synthetic and real data backup ourtheoretical findings.

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data