English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) power many state-of-the-art systems in naturallanguage processing. However, these models are extremely computationallyexpensive, even at inference time, raising the natural question: when is theextra cost of deploying a larger model worth the anticipated boost incapabilities? Better understanding this tradeoff fundamentally could benefitfrom an inference efficiency metric that is both (i) easily comparable acrossmodels from different providers, and (ii) representative of the true cost ofrunning queries in an isolated performance environment. Unfortunately, accessto LLMs today is largely restricted to black-box text generation APIs and rawruntimes measured through this interface do not satisfy these desiderata: modelproviders can apply various software and hardware optimizations orthogonal tothe model, and models served on shared infrastructure are susceptible toperformance contention. To circumvent these problems, we propose a new metricfor comparing inference efficiency across models. This metric puts models onequal footing as though they were served (i) on uniform hardware and software,and (ii) without performance contention. We call this metric the\emph{idealized runtime}, and we propose a methodology to efficiently estimatethis metric for autoregressive Transformer models. We also propose cost-awarevariants that incorporate the number of accelerators needed to serve the model.Using these metrics, we compare ten state-of-the-art LLMs to provide the firstanalysis of inference efficiency-capability tradeoffs; we make severalobservations from this analysis, including the fact that the superior inferenceruntime performance of certain APIs is often a byproduct of optimizationswithin the API rather than the underlying model. Our methodology alsofacilitates the efficient comparison of different software and hardware stacks.

Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs