English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) are routinely pre-trained on billions of tokens,only to restart the process over again once new data becomes available. A muchcheaper and more efficient solution would be to enable the continualpre-training of these models, i.e. updating pre-trained models with new datainstead of re-training them from scratch. However, the distribution shiftinduced by novel data typically results in degraded performance on past data.Taking a step towards efficient continual pre-training, in this work, weexamine the effect of different warm-up strategies. Our hypothesis is that thelearning rate must be re-increased to improve compute efficiency when trainingon a new dataset. We study the warmup phase of models pre-trained on the Pile(upstream data, 300B tokens) as we continue to pre-train on SlimPajama(downstream data, 297B tokens), following a linear warmup and cosine decayschedule. We conduct all experiments on the Pythia 410M language modelarchitecture and evaluate performance through validation perplexity. Weexperiment with different pre-training checkpoints, various maximum learningrates, and various warmup lengths. Our results show that while rewarming modelsfirst increases the loss on upstream and downstream data, in the longer run itimproves the downstream performance, outperforming models trained fromscratch$\unicode{x2013}$even for a large downstream dataset.

Continual Pre-Training of Large Language Models: How to (re)warm your model?