English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent works on over-parameterized neural networks have shown that thestochasticity in optimizers has the implicit regularization effect ofminimizing the sharpness of the loss function (in particular, the trace of itsHessian) over the family zero-loss solutions. More explicit forms of flatnessregularization also empirically improve the generalization performance.However, it remains unclear why and when flatness regularization leads tobetter generalization. This work takes the first step toward understanding theinductive bias of the minimum trace of the Hessian solutions in an importantsetting: learning deep linear networks from linear measurements, also known as\emph{deep matrix factorization}. We show that for all depth greater than one,with the standard Restricted Isometry Property (RIP) on the measurements,minimizing the trace of Hessian is approximately equivalent to minimizing theSchatten 1-norm of the corresponding end-to-end matrix parameters (i.e., theproduct of all layer matrices), which in turn leads to better generalization.We empirically verify our theoretical findings on synthetic datasets.

The Inductive Bias of Flatness Regularization for Deep Matrix Factorization