English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Cohen's and Fleiss' kappa are well-known measures for inter-raterreliability. However, they only allow a rater to select exactly one categoryfor each subject. This is a severe limitation in some research contexts: forexample, measuring the inter-rater reliability of a group of psychiatristsdiagnosing patients into multiple disorders is impossible with these measures.This paper proposes a generalisation of the Fleiss' kappa coefficient thatlifts this limitation. Specifically, the proposed $\kappa$ statistic measuresinter-rater reliability between multiple raters classifying subjects intoone-or-more nominal categories. These categories can be weighted according totheir importance, and the measure can take into account the category hierarchy(e.g., categories consisting of subcategories that are only available whenchoosing the main category like a primary psychiatric disorder andsub-disorders; but much more complex dependencies between categories arepossible as well). The proposed $\kappa$ statistic can handle missing data anda varying number of raters for subjects or categories. The paper brieflyoverviews existing methods allowing raters to classify subjects into multiplecategories. Next, we derive our proposed measure step-by-step and prove thatthe proposed measure equals Fleiss' kappa when a fixed number of raters choseone category for each subject. The measure was developed to investigate thereliability of a new mathematics assessment method, of which an example iselaborated. The paper concludes with the worked-out example of psychiatristsdiagnosing patients into multiple disorders.

Measuring agreement among several raters classifying subjects into one-or-more (hierarchical) nominal categories. A generalisation of Fleiss' kappa