English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, Moffat et al. proposed an analytic framework, namely C/W/L/A, foroffline evaluation metrics. This framework allows information retrieval (IR)researchers to design evaluation metrics through the flexible combination ofuser browsing models and user gain aggregations. However, the statisticalstability of C/W/L/A metrics with different aggregations is not yetinvestigated. In this study, we investigate the statistical stability ofC/W/L/A metrics from the perspective of: (1) the system ranking similarityamong aggregations, (2) the system ranking consistency of aggregations and (3)the discriminative power of aggregations. More specifically, we combinedvarious aggregation functions with the browsing model of Precision, DiscountedCumulative Gain (DCG), Rank-Biased Precision (RBP), INST, Average Precision(AP) and Expected Reciprocal Rank (ERR), examing their performances in terms ofsystem ranking similarity, system ranking consistency and discriminative poweron two offline test collections. Our experimental result suggests that, interms of system ranking consistency and discriminative power, the aggregationfunction of expected rate of gain (ERG) has an outstanding performance whilethe aggregation function of maximum relevance usually has an insufficientperformance. The result also suggests that Precision, DCG, RBP, INST and APwith their canonical aggregation all have favourable performances in systemranking consistency and discriminative power; but for ERR, replacing itscanonical aggregation with ERG can further strengthen the discriminative powerwhile obtaining a system ranking list similar to the canonical version at thesame time.

A Meta-Evaluation of C/W/L/A Metrics: System Ranking Similarity, System Ranking Consistency and Discriminative Power