English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Benchmarks such as GLUE have helped drive advances in NLP by incentivizingthe creation of more accurate models. While this leaderboard paradigm has beenremarkably successful, a historical focus on performance-based evaluation hasbeen at the expense of other qualities that the NLP community values in models,such as compactness, fairness, and energy efficiency. In this opinion paper, westudy the divergence between what is incentivized by leaderboards and what isuseful in practice through the lens of microeconomic theory. We frame both theleaderboard and NLP practitioners as consumers and the benefit they get from amodel as its utility to them. With this framing, we formalize how leaderboards-- in their current form -- can be poor proxies for the NLP community at large.For example, a highly inefficient model would provide less utility topractitioners but not to a leaderboard, since it is a cost that only the formermust bear. To allow practitioners to better estimate a model's utility to them,we advocate for more transparency on leaderboards, such as the reporting ofstatistics that are of practical concern (e.g., model size, energy efficiency,and inference latency).

Utility is in the Eye of the User: A Critique of NLP Leaderboards