English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Automatic metrics play a crucial role in machine translation. Despite thewidespread use of n-gram-based metrics, there has been a recent surge in thedevelopment of pre-trained model-based metrics that focus on measuring sentencesemantics. However, these neural metrics, while achieving higher correlationswith human evaluations, are often considered to be black boxes with potentialbiases that are difficult to detect. In this study, we systematically analyzeand compare various mainstream and cutting-edge automatic metrics from theperspective of their guidance for training machine translation systems. ThroughMinimum Risk Training (MRT), we find that certain metrics exhibit robustnessdefects, such as the presence of universal adversarial translations in BLEURTand BARTScore. In-depth analysis suggests two main causes of these robustnessdeficits: distribution biases in the training datasets, and the tendency of themetric paradigm. By incorporating token-level constraints, we enhance therobustness of evaluation metrics, which in turn leads to an improvement in theperformance of machine translation systems. Codes are available at\url{https://github.com/powerpuffpomelo/fairseq_mrt}.

BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training