English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

High-concurrency asynchronous training upon parameter server (PS)architecture and high-performance synchronous training upon all-reduce (AR)architecture are the most commonly deployed distributed training modes forrecommendation models. Although synchronous AR training is designed to havehigher training efficiency, asynchronous PS training would be a better choicefor training speed when there are stragglers (slow workers) in the sharedcluster, especially under limited computing resources. An ideal way to takefull advantage of these two training modes is to switch between them upon thecluster status. However, switching training modes often requires tuninghyper-parameters, which is extremely time- and resource-consuming. We find twoobstacles to a tuning-free approach: the different distribution of the gradientvalues and the stale gradients from the stragglers. This paper proposes GlobalBatch gradients Aggregation (GBA) over PS, which aggregates and appliesgradients with the same global batch size as the synchronous training. Atoken-control process is implemented to assemble the gradients and decay thegradients with severe staleness. We provide the convergence analysis to revealthat GBA has comparable convergence properties with the synchronous training,and demonstrate the robustness of GBA the recommendation models against thegradient staleness. Experiments on three industrial-scale recommendation tasksshow that GBA is an effective tuning-free approach for switching. Compared tothe state-of-the-art derived asynchronous training, GBA achieves up to 0.2%improvement on the AUC metric, which is significant for the recommendationmodels. Meanwhile, under the strained hardware resource, GBA speeds up at least2.4x compared to synchronous training.

GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Model