English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Moffat recently commented on our previous work. Our work focused on howlaying the foundations of our evaluation methodology into the theory ofmeasurement can improve our knowledge and understanding of the evaluationmeasures we use in IR and how it can shed light on the different types ofscales adopted by our evaluation measures; we also provided evidence, throughextensive experimentation, on the impact of the different types of scales onthe statistical analyses, as well as on the impact of departing from theirassumptions. Moreover, we investigated, for the first time in IR, the conceptof meaningfulness, i.e. the invariance of the experimental statements andinferences you draw, and proposed it as a way to ensure more valid andgeneralizabile results. Moffat's comments build on: (i) misconceptions aboutthe representational theory of measurement, such as what an interval scaleactually is and what axioms it has to comply with; (ii) they totally miss thecentral concept of meaningfulness. Therefore, we reply to Moffat's comments byproperly framing them in the representational theory of measurement and in theconcept of meaningfulness. All in all, we can only reiterate what we saidseveral times: the goal of this research line is to theoretically ground ourevaluation methodology - and IR is a field where it is extremely challenging toperform any theoretical advances - in order to aim for more robust andgeneralizable inferences - something we currently lack in the field. Possiblythere are other and better ways to achieve this objective and these proposalscould emerge from an open discussion in the field and from the work of others.On the other hand, reducing everything to a contrast on what is (or pretend tobe) an interval scale or whether all or none evaluation measures are intervalscales may be more a barrier from than a help in progressing towards this goal.

Response to Moffat's Comment on "Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales"