English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While test collections provide the cornerstone for Cranfield-based evaluationof information retrieval (IR) systems, it has become practically infeasible torely on traditional pooling techniques to construct test collections at thescale of today's massive document collections. In this paper, we propose a newintelligent topic selection method which reduces the number of search topicsneeded for reliable IR evaluation. To rigorously assess our method, weintegrate previously disparate lines of research on intelligent topic selectionand deep vs. shallow judging. While prior work on intelligent topic selectionhas never been evaluated against shallow judging baselines, prior work on deepvs. shallow judging has largely argued for shallowed judging, but assumingrandom topic selection. We argue that for evaluating any topic selectionmethod, ultimately one must ask whether it is actually useful to select topics,or should one simply perform shallow judging over many topics? In seeking arigorous answer to this over-arching question, we conduct a comprehensiveinvestigation over a set of relevant factors never previously studied together1) topic selection method 2) the effect of topic familiarity on human judgingspeed and 3) how different topic generation processes impact (i) budgetutilization and (ii) the resultant quality of judgments. Experiments on NISTTREC Robust 2003 and Robust 2004 test collections show that not only can wereliably evaluate IR systems with fewer topics, but also that 1) when topicsare intelligently selected, deep judging is often more cost-effective thanshallow judging in evaluation reliability and 2) topic familiarity and topicgeneration costs greatly impact the evaluation cost vs. reliability trade-off.Our findings challenge conventional wisdom in showing that deep judging isoften preferable to shallow judging when topics are selected intelligently.

Intelligent Topic Selection for Low-Cost Information Retrieval Evaluation: A New Perspective on Deep vs. Shallow Judging