English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, it was shown that most popular IR measures are not interval-scaled,implying that decades of experimental IR research used potentially impropermethods, which may have produced questionable results. However, it was unclearif and to what extent these findings apply to actual evaluations and thisopened a debate in the community with researchers standing on oppositepositions about whether this should be considered an issue (or not) and to whatextent.  In this paper, we first give an introduction to the representationalmeasurement theory explaining why certain operations and significance tests arepermissible only with scales of a certain level. For that, we introduce thenotion of meaningfulness specifying the conditions under which the truth (orfalsity) of a statement is invariant under permissible transformations of ascale. Furthermore, we show how the recall base and the length of the run maymake comparison and aggregation across topics problematic.  Then we propose a straightforward and powerful approach for turning anevaluation measure into an interval scale, and describe an experimentalevaluation of the differences between using the original measures and theinterval-scaled ones.  For all the regarded measures - namely Precision, Recall, Average Precision,(Normalized) Discounted Cumulative Gain, Rank-Biased Precision and ReciprocalRank - we observe substantial effects, both on the order of average values andon the outcome of significance tests. For the latter, previously significantdifferences turn out to be insignificant, while insignificant ones becomesignificant. The effect varies remarkably between the tests considered butoverall, on average, we observed a 25% change in the decision about whichsystems are significantly different and which are not.

Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales