English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Training neural networks with many processors can reduce time-to-solution;however, it is challenging to maintain convergence and efficiency at largescales. The Kronecker-factored Approximate Curvature (K-FAC) was recentlyproposed as an approximation of the Fisher Information Matrix that can be usedin natural gradient optimizers. We investigate here a scalable K-FAC design andits applicability in convolutional neural network (CNN) training at scale. Westudy optimization techniques such as layer-wise distribution strategies,inverse-free second-order gradient evaluation, and dynamic K-FAC updatedecoupling to reduce training time while preserving convergence. We useresidual neural networks (ResNet) applied to the CIFAR-10 and ImageNet-1kdatasets to evaluate the correctness and scalability of our K-FAC gradientpreconditioner. With ResNet-50 on the ImageNet-1k dataset, our distributedK-FAC implementation converges to the 75.9% MLPerf baseline in 18-25% less timethan does the classic stochastic gradient descent (SGD) optimizer across scaleson a GPU cluster.

Convolutional Neural Network Training with Distributed K-FAC