English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Understanding the implicit regularization imposed by neural networkarchitectures and gradient based optimization methods is a key challenge indeep learning and AI. In this work we provide sharp results for the implicitregularization imposed by the gradient flow of Diagonal Linear Networks (DLNs)in the over-parameterized regression setting and, potentially surprisingly,link this to the phenomenon of phase transitions in generalized hardness ofapproximation (GHA). GHA generalizes the phenomenon of hardness ofapproximation from computer science to, among others, continuous and robustoptimization. It is well-known that the $\ell^1$-norm of the gradient flow ofDLNs with tiny initialization converges to the objective function of basispursuit. We improve upon these results by showing that the gradient flow ofDLNs with tiny initialization approximates minimizers of the basis pursuitoptimization problem (as opposed to just the objective function), and we obtainnew and sharp convergence bounds w.r.t.\ the initialization size. Non-sharpnessof our results would imply that the GHA phenomenon would not occur for thebasis pursuit optimization problem -- which is a contradiction -- thus implyingsharpness. Moreover, we characterize $\textit{which}$ $\ell_1$ minimizer of thebasis pursuit problem is chosen by the gradient flow whenever the minimizer isnot unique. Interestingly, this depends on the depth of the DLN.

Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networks