English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

As large language models (LLMs) continue to advance, accurately andcomprehensively evaluating their performance becomes increasingly challenging.Ranking the relative performance of LLMs based on Elo ratings, according tohuman judgment, is gaining more popularity. However, the extent to which humansand LLMs are capable evaluators remains uncertain. This study investigates thebehavior of crowd-sourced and expert annotators, as well as LLMs, whencomparing outputs from different models. To achieve this, we curate a datasetof intentionally flawed machine-generated answers. Our findings reveal aconcerning bias in the evaluation process, as answers with factual errors arerated more favorably than answers that are too short or contained grammaticalerrors. To address this issue, we propose independently evaluatingmachine-generated text across multiple dimensions, rather than merging all theevaluation aspects into a single score. We instantiate this idea with the Elorating system, resulting in the Multi-Elo Rating System (MERS). Empiricalresults from our study reveal that this proposed approach significantlyenhances the quality of LLM-based evaluations, particularly in terms of factualaccuracy. However, there is no significant improvement in crowd-sourced-basedevaluations, indicating the need for further investigation.

Style Over Substance: Evaluation Biases for Large Language Models