English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In recent years, large pre-trained Transformer-based language models have ledto dramatic improvements in many natural language understanding tasks. To trainthese models with increasing sizes, many neural network practitioners attemptto increase the batch sizes in order to leverage multiple GPUs to improvetraining speed. However, increasing the batch size often makes the optimizationmore difficult, leading to slow convergence or poor generalization that canrequire orders of magnitude more training time to achieve the same modelquality. In this paper, we explore the steepness of the loss landscape oflarge-batch optimization for adapting pre-trained Transformer-based languagemodels to domain-specific tasks and find that it tends to be highly complex andirregular, posing challenges to generalization on downstream tasks.  To tackle this challenge, we propose ScaLA, a novel and efficient method toaccelerate the adaptation speed of pre-trained transformer networks. Differentfrom prior methods, we take a sequential game-theoretic approach by addinglightweight adversarial noise into large-batch optimization, whichsignificantly improves adaptation speed while preserving model generalization.Experiment results show that ScaLA attains 2.7--9.8$\times$ adaptation speedupsover the baseline for GLUE on BERT-base and RoBERTa-large, while achievingcomparable and sometimes higher accuracy than the state-of-the-art large-batchoptimization methods. Finally, we also address the theoretical aspect oflarge-batch optimization with adversarial noise and provide a theoreticalconvergence rate analysis for ScaLA using techniques for analyzing non-convexsaddle-point problems.

ScaLA: Accelerating Adaptation of Pre-Trained Transformer-Based Language Models via Efficient Large-Batch Adversarial Noise