English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Matrix factorization is a simple and natural test-bed to investigate theimplicit regularization of gradient descent. Gunasekar et al. (2017)conjectured that Gradient Flow with infinitesimal initialization converges tothe solution that minimizes the nuclear norm, but a series of recent papersargued that the language of norm minimization is not sufficient to give a fullcharacterization for the implicit regularization. In this work, we providetheoretical and empirical evidence that for depth-2 matrix factorization,gradient flow with infinitesimal initialization is mathematically equivalent toa simple heuristic rank minimization algorithm, Greedy Low-Rank Learning, undersome reasonable assumptions. This generalizes the rank minimization view fromprevious works to a much broader setting and enables us to constructcounter-examples to refute the conjecture from Gunasekar et al. (2017). We alsoextend the results to the case where depth $\ge 3$, and we show that thebenefit of being deeper is that the above convergence has a much weakerdependence over initialization magnitude so that this rank minimization is morelikely to take effect for initialization with practical scale.

Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning