English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Online ranker evaluation is one of the key challenges in informationretrieval. While the preferences of rankers can be inferred by interleavingmethods, the problem of how to effectively choose the ranker pair thatgenerates the interleaved list without degrading the user experience too muchis still challenging. On the one hand, if two rankers have not been comparedenough, the inferred preference can be noisy and inaccurate. On the other, iftwo rankers are compared too many times, the interleaving process inevitablyhurts the user experience too much. This dilemma is known as the explorationversus exploitation tradeoff. It is captured by the $K$-armed dueling banditproblem, which is a variant of the $K$-armed bandit problem, where the feedbackcomes in the form of pairwise preferences. Today's deployed search systems canevaluate a large number of rankers concurrently, and scaling effectively in thepresence of numerous rankers is a critical aspect of $K$-armed dueling banditproblems.  In this paper, we focus on solving the large-scale online ranker evaluationproblem under the so-called Condorcet assumption, where there exists an optimalranker that is preferred to all other rankers. We propose Merge Double ThompsonSampling (MergeDTS), which first utilizes a divide-and-conquer strategy thatlocalizes the comparisons carried out by the algorithm to small batches ofrankers, and then employs Thompson Sampling (TS) to reduce the comparisonsbetween suboptimal rankers inside these small batches. The effectiveness(regret) and efficiency (time complexity) of MergeDTS are extensively evaluatedusing examples from the domain of online evaluation for web search. Our mainfinding is that for large-scale Condorcet ranker evaluation problems, MergeDTSoutperforms the state-of-the-art dueling bandit algorithms.

MergeDTS: A Method for Effective Large-Scale Online Ranker Evaluation