English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We investigate large language model performance across five orders ofmagnitude of compute scaling in eleven recent model architectures. We show thataverage benchmark performance, aggregating over many individual tasks andevaluations as in the commonly-used BIG-Bench dataset, is decently predictableas a function of training compute scale. Specifically, when extrapolatingBIG-Bench Hard performance across one order of magnitude in compute, we observeaverage absolute errors of 6 percentage points (pp). By contrast, extrapolationfor individual BIG-Bench tasks across an order of magnitude in compute yieldshigher average errors of 18pp. Nonetheless, individual task performance remainssignificantly more predictable than chance. Overall, our work suggests computescaling provides a promising basis to forecast AI capabilities in diversebenchmarks, though predicting performance in specific tasks poses challenges.

How predictable is language model benchmark performance?