English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Using quasi-Newton methods in stochastic optimization is not a trivial task.In deterministic optimization, these methods are often a common choice due totheir excellent performance regardless of the problem's condition number.However, standard quasi-Newton methods fail to extract curvature informationfrom noisy gradients in stochastic optimization. Moreover, pre-conditioningnoisy gradient observations tend to amplify the noise by a factor given by thelargest eigenvalue of the pre-conditioning matrix. We propose a Bayesianapproach to obtain a Hessian matrix approximation for stochastic optimizationthat minimizes the secant equations residue while retaining the smallesteigenvalue above a specified limit. Thus, the proposed approach assistsstochastic gradient descent to converge to local minima without augmentinggradient noise. The prior distribution is modeled as the exponential of thenegative squared distance to the previous Hessian approximation, in theFrobenius sense, with logarithmic barriers imposing extreme eigenvaluesconstraints. The likelihood distribution is derived from the secant equations,i.e., the probability of obtaining the observed gradient evaluations for agiven candidate Hessian approximation. We propose maximizing the log-posteriorusing the Newton-CG method. Numerical results on a stochastic quadraticfunction and a $\ell_2$ regularized logistic regression problem are presented.In all the cases tested, our approach improves the convergence of stochasticgradient descent, compensating for the overhead of solving the log-posteriormaximization. In particular, pre-conditioning the stochastic gradient with theinverse of our Hessian approximation becomes more advantageous the larger thecondition number of the problem is.

Approximating Hessian matrices using Bayesian inference: a new approach for quasi-Newton methods in stochastic optimization