English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Inter-rater reliability (IRR) is one of the commonly used tools for assessingthe quality of ratings from multiple raters as it is easily obtainable from theobserved ratings themselves. However, applicant selection procedures based onratings from multiple raters usually result in a binary outcome; the applicantis either selected or not. This final outcome is not considered in IRR, whichinstead focuses on the ratings of the individual subjects or objects. In thiswork, we outline the connection between the ratings' measurement model (usedfor IRR) and a binary classification framework. We develop a quantileapproximation which allows us to estimate the probability of correctlyselecting the best applicants and compute error probabilities of the selectionprocedure (i.e., false-positive and false-negative rate) under the assumptionof the ratings' validity. If the ratings are not completely valid, the computederror probabilities correspond to a lower bound on the true errorprobabilities. We draw connections between the inter-rater reliability and thebinary classification metrics, showing that binary classification metricsdepend solely on the IRR coefficient and proportion of selected applicants. Weassess the performance of the quantile approximation in a simulation study andapply it in an example comparing the reliability of multiple grant peer reviewselection procedures.

Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability