English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently there has been significant theoretical progress on understanding theconvergence and generalization of gradient-based methods on nonconvex losseswith overparameterized models. Nevertheless, many aspects of optimization andgeneralization and in particular the critical role of small randominitialization are not fully understood. In this paper, we take a step towardsdemystifying this role by proving that small random initialization followed bya few iterations of gradient descent behaves akin to popular spectral methods.We also show that this implicit spectral bias from small random initialization,which is provably more prominent for overparameterized models, also puts thegradient descent iterations on a particular trajectory towards solutions thatare not only globally optimal but also generalize well. Concretely, we focus onthe problem of reconstructing a low-rank matrix from a few measurements via anatural nonconvex formulation. In this setting, we show that the trajectory ofthe gradient descent iterations from small random initialization can beapproximately decomposed into three phases: (I) a spectral or alignment phasewhere we show that that the iterates have an implicit spectral bias akin tospectral initialization allowing us to show that at the end of this phase thecolumn space of the iterates and the underlying low-rank matrix aresufficiently aligned, (II) a saddle avoidance/refinement phase where we showthat the trajectory of the gradient iterates moves away from certain degeneratesaddle points, and (III) a local refinement phase where we show that afteravoiding the saddles the iterates converge quickly to the underlying low-rankmatrix. Underlying our analysis are insights for the analysis ofoverparameterized nonconvex optimization schemes that may have implications forcomputational problems beyond low-rank reconstruction.

Small random initialization is akin to spectral learning: Optimization  and generalization guarantees for overparameterized low-rank matrix  reconstruction