English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Hessian-free training has become a popular parallel second or- deroptimization technique for Deep Neural Network training. This study aims atspeeding up Hessian-free training, both by means of decreasing the amount ofdata used for training, as well as through reduction of the number of Krylovsubspace solver iterations used for implicit estimation of the Hessian. In thispaper, we develop an L-BFGS based preconditioning scheme that avoids the needto access the Hessian explicitly. Since L-BFGS cannot be regarded as afixed-point iteration, we further propose the employment of flexible Krylovsubspace solvers that retain the desired theoretical convergence guarantees oftheir conventional counterparts. Second, we propose a new sampling algorithm,which geometrically increases the amount of data utilized for gradient andKrylov subspace iteration calculations. On a 50-hr English Broadcast News task,we find that these methodologies provide roughly a 1.5x speed-up, whereas, on a300-hr Switchboard task, these techniques provide over a 2.3x speedup, with noloss in WER. These results suggest that even further speed-up is expected, asproblems scale and complexity grows.

Accelerating Hessian-free optimization for deep neural networks by implicit preconditioning and sampling