English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Efficiency is a key property to foster inclusiveness and reduce environmentalcosts, especially in an era of LLMs. In this work, we provide a comprehensiveevaluation of efficiency for MT evaluation metrics. Our approach involvesreplacing computation-intensive transformers with lighter alternatives andemploying linear and quadratic approximations for alignment algorithms on topof LLM representations. We evaluate six (reference-free and reference-based)metrics across three MT datasets and examine 16 lightweight transformers. Inaddition, we look into the training efficiency of metrics like COMET byutilizing adapters. Our results indicate that (a) TinyBERT provides the optimalbalance between quality and efficiency, (b) CPU speed-ups are more substantialthan those on GPU; (c) WMD approximations yield no efficiency gains whilereducing quality and (d) adapters enhance training efficiency (regardingbackward pass speed and memory requirements) as well as, in some cases, metricquality. These findings can help to strike a balance between evaluation speedand quality, which is essential for effective NLG systems. Furthermore, ourresearch contributes to the ongoing efforts to optimize NLG evaluation metricswith minimal impact on performance. To our knowledge, ours is the mostcomprehensive analysis of different aspects of efficiency for MT metricsconducted so far.

EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics