English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Collecting human judgements is currently the most reliable evaluation methodfor natural language generation systems. Automatic metrics have reported flawswhen applied to measure quality aspects of generated text and have been shownto correlate poorly with human judgements. However, human evaluation is timeand cost-intensive, and we lack consensus on designing and conducting humanevaluation experiments. Thus there is a need for streamlined approaches forefficient collection of human judgements when evaluating natural languagegeneration systems. Therefore, we present a dynamic approach to measure therequired number of human annotations when evaluating generated outputs inrelative comparison settings. We propose an agent-based framework of humanevaluation to assess multiple labelling strategies and methods to decide thebetter model in a simulation and a crowdsourcing case study. The main resultsindicate that a decision about the superior model can be made with highprobability across different labelling strategies, where assigning a singlerandom worker per task requires the least overall labelling effort and thus theleast cost.

Dynamic Human Evaluation for Relative Model Comparisons