English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The second-order optimization methods, notably the D-KFAC (DistributedKronecker Factored Approximate Curvature) algorithms, have gained traction onaccelerating deep neural network (DNN) training on GPU clusters. However,existing D-KFAC algorithms require to compute and communicate a large volume ofsecond-order information, i.e., Kronecker factors (KFs), before preconditioninggradients, resulting in large computation and communication overheads as wellas a high memory footprint. In this paper, we propose DP-KFAC, a noveldistributed preconditioning scheme that distributes the KF constructing tasksat different DNN layers to different workers. DP-KFAC not only retains theconvergence property of the existing D-KFAC algorithms but also enables threebenefits: reduced computation overhead in constructing KFs, no communication ofKFs, and low memory footprint. Extensive experiments on a 64-GPU cluster showthat DP-KFAC reduces the computation overhead by 1.55x-1.65x, the communicationcost by 2.79x-3.15x, and the memory footprint by 1.14x-1.47x in eachsecond-order update compared to the state-of-the-art D-KFAC methods.

Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning