Statistical Models for Repeated Categorical Ratings: The R Package rater

A common problem in many disciplines is the need to assign a set of itemsinto categories or classes with known labels. This is often done by one or moreexpert raters, or sometimes by an automated process. If these assignments or`ratings' are difficult to make accurately, a common tactic is to repeat themby different raters, or even by the same rater multiple times on differentoccasions. We present an R package `rater`, available on CRAN, that implementsBayesian versions of several statistical models for analysis of repeatedcategorical rating data. Inference is possible for the true underlying (latent)class of each item, as well as the accuracy of each rater. The models areextensions of, and include, the Dawid-Skene model, and we implemented themusing the Stan probabilistic programming language. We illustrate the use of`rater` through a few examples. We also discuss in detail the techniques ofmarginalisation and conditioning, which are necessary for these models but alsoapply more generally to other models implemented in Stan.