English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Hessians of neural network (NN) contain essential information about thecurvature of NN loss landscapes which can be used to estimate NN generalizationcapabilities. We have previously proposed generalization criteria that rely onthe observation that Hessian eigenvalue spectral density (HESD) behavessimilarly for a wide class of NNs. This paper further studies theirapplicability by investigating factors that can result in different types ofHESD. We conduct a wide range of experiments showing that HESD mainly haspositive eigenvalues (MP-HESD) for NN training and fine-tuning with variousoptimizers on different datasets with different preprocessing and augmentationprocedures. We also show that mainly negative HESD (MN-HESD) is a consequenceof external gradient manipulation, indicating that the previously proposedHessian analysis methodology cannot be applied in such cases. We also proposecriteria and corresponding conditions to determine HESD type and estimate NNgeneralization potential. These HESD types and previously proposedgeneralization criteria are combined into a unified HESD analysis methodology.Finally, we discuss how HESD changes during training, and show the occurrenceof quasi-singular (QS) HESD and its influence on the proposed methodology andon the conventional assumptions about the relation between Hessian eigenvaluesand NN loss landscape curvature.

The effects of Hessian eigenvalue spectral density type on the  applicability of Hessian analysis to generalization capability assessment of  neural networks