English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural information retrieval (IR) systems have progressed rapidly in recentyears, in large part due to the release of publicly available benchmarkingtasks. Unfortunately, some dimensions of this progress are illusory: themajority of the popular IR benchmarks today focus exclusively on downstreamtask accuracy and thus conceal the costs incurred by systems that trade awayefficiency for quality. Latency, hardware cost, and other efficiencyconsiderations are paramount to the deployment of IR systems in user-facingsettings. We propose that IR benchmarks structure their evaluation methodologyto include not only metrics of accuracy, but also efficiency considerationssuch as a query latency and the corresponding cost budget for a reproduciblehardware setting. For the popular IR benchmarks MS MARCO and XOR-TyDi, we showhow the best choice of IR system varies according to how these efficiencyconsiderations are chosen and weighed. We hope that future benchmarks willadopt these guidelines toward more holistic IR evaluation.

Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking