English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Second-order optimizers hold intriguing potential for deep learning, butsuffer from increased cost and sensitivity to the non-convexity of the losssurface as compared to gradient-based approaches. We introduce a coordinatedescent method to train deep neural networks for classification tasks thatexploits global convexity of the cross-entropy loss in the weights of thelinear layer. Our hybrid Newton/Gradient Descent (NGD) method is consistentwith the interpretation of hidden layers as providing an adaptive basis and thelinear layer as providing an optimal fit of the basis to data. By alternatingbetween a second-order method to find globally optimal parameters for thelinear layer and gradient descent to train the hidden layers, we ensure anoptimal fit of the adaptive basis to data throughout training. The size of theHessian in the second-order step scales only with the number weights in thelinear layer and not the depth and width of the hidden layers; furthermore, theapproach is applicable to arbitrary hidden layer architecture. Previous workapplying this adaptive basis perspective to regression problems demonstratedsignificant improvements in accuracy at reduced training cost, and this workcan be viewed as an extension of this approach to classification problems. Wefirst prove that the resulting Hessian matrix is symmetric semi-definite, andthat the Newton step realizes a global minimizer. By studying classification ofmanufactured two-dimensional point cloud data, we demonstrate both animprovement in validation error and a striking qualitative difference in thebasis functions encoded in the hidden layer when trained using NGD. Applicationto image classification benchmarks for both dense and convolutionalarchitectures reveals improved training accuracy, suggesting possible gains ofsecond-order methods over gradient descent.

A block coordinate descent optimizer for classification problems exploiting convexity