English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

As language models scale up, it becomes increasingly expensive to verifyresearch ideas because conclusions on small models do not trivially transfer tolarge ones. A possible solution is to establish a generic system thataccurately predicts certain metrics for large models without training them.Existing scaling laws require hyperparameter search on the largest models,limiting their predicative capability. In this paper, we present an approach(namely {\mu}Scaling) to predict the pre-training loss, based on ourobservations that Maximal Update Parametrization ({\mu}P) enables accuratefitting of scaling laws close to common loss basins in hyperparameter space.With {\mu}Scaling, different model designs can be compared on large scales bytraining only their smaller counterparts. Further, we introduce nanoLM: anaffordable LLM pre-training benchmark that facilitates this new researchparadigm. With around 14% of the one-time pre-training cost, we can accuratelyforecast the loss for models up to 52B. Our goal with nanoLM is to empowerresearchers with limited resources to reach meaningful conclusions on largemodels. We also aspire for our benchmark to serve as a bridge between theacademic community and the industry. Code for {\mu}Scaling is available athttps://github.com/cofe-ai/Mu-scaling. Code for nanoLLM will be availablelater.

nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales