English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Kendall's tau is frequently used to meta-evaluate how well machinetranslation (MT) evaluation metrics score individual translations. Its focus onpairwise score comparisons is intuitive but raises the question of how tiesshould be handled, a gray area that has motivated different variants in theliterature. We demonstrate that, in settings like modern MT meta-evaluation,existing variants have weaknesses arising from their handling of ties, and insome situations can even be gamed. We propose instead to meta-evaluate metricswith a version of pairwise accuracy that gives metrics credit for correctlypredicting ties, in combination with a tie calibration procedure thatautomatically introduces ties into metric scores, enabling fair comparisonbetween metrics that do and do not predict ties. We argue and provideexperimental evidence that these modifications lead to fairer ranking-basedassessments of metric performance.

Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration