Premium
RRep: A composite index to assess and test rater precision
Author(s) -
Vanacore Amalia,
Pellegrino Maria Sole
Publication year - 2018
Publication title -
quality and reliability engineering international
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.913
H-Index - 62
eISSN - 1099-1638
pISSN - 0748-8017
DOI - 10.1002/qre.2334
Subject(s) - benchmarking , nonparametric statistics , computer science , index (typography) , monte carlo method , composite index , test (biology) , reliability engineering , inter rater reliability , process (computing) , psychology , statistics , rating scale , composite indicator , econometrics , engineering , mathematics , management , paleontology , world wide web , economics , biology , operating system
In subjective evaluation systems, raters act as measurement instruments providing useful evaluations for taking strategic and/or operational decisions. The assessment of rater evaluative ability in terms of accuracy and precision is of critical importance since rater unreliability may compromise the quality of the decision‐making process. The focus of this paper is on rater precision: we propose a novel composite index to assess the rater ability to provide evaluations repeatable over time and reproducible over scales. The extent of rater precision is qualified via a nonparametric benchmarking procedure. The properties of both proposed index and benchmarking procedure have been analysed via a Monte Carlo simulation study.