English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

First-order optimization methods remain the standard for training deep neuralnetworks (DNNs). Optimizers like Adam incorporate limited curvature informationby preconditioning the stochastic gradient with a diagonal matrix. Despite thewidespread adoption of first-order methods, second-order optimizationalgorithms often exhibit superior convergence compared to methods like Adam andSGD. However, their practicality in training DNNs is still limited by asignificantly higher per-iteration computational cost compared to first-ordermethods. In this thesis, we present AdaFisher, a novel adaptive second-orderoptimizer that leverages a diagonal block-Kronecker approximation of the Fisherinformation matrix to adaptively precondition gradients. AdaFisher aims tobridge the gap between the improved convergence and generalization ofsecond-order methods and the computational efficiency needed for training DNNs.Despite the traditionally slower speed of second-order optimizers, AdaFisher iseffective for tasks such as image classification and language modeling,exhibiting remarkable stability and robustness during hyperparameter tuning. Wedemonstrate that AdaFisher outperforms state-of-the-art optimizers in bothaccuracy and convergence speed. The code is available fromhttps://github.com/AtlasAnalyticsLab/AdaFisher.

Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis