Open AccessCurmElo: The theory and practice of a forced-choice approach to producing preference rankings
Author(s) -
Soham Sankaran,
Jacob Derechin,
Nicholas A. Christakis
Publication year - 2021
Publication title -
plos one
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.99
H-Index - 332
ISSN - 1932-6203
DOI - 10.1371/journal.pone.0252145
Subject(s) - preference , nonsense , ranking (information retrieval) , population , set (abstract data type) , computer science , preference learning , social choice theory , revealed preference , identifier , preference theory , psychology , natural language processing , information retrieval , econometrics , statistics , mathematics , mathematical economics , sociology , demography , biochemistry , chemistry , gene , programming language
We introduce CurmElo, a forced-choice approach to producing a preference ranking of an arbitrary set of objects that combines the Elo algorithm with novel techniques for detecting and correcting for (1) preference heterogeneity induced polarization in preferences among raters, and (2) intransitivity in preference rankings. We detail the application of CurmElo to the problem of generating approximately preference-neutral identifiers, in this case four-letter and five-letter nonsense words patterned on the phonological conventions of the English language, using a population of Amazon Mechanical Turk workers. We find evidence that human raters have significant non-uniform preferences over these nonsense words, and we detail the consequences of this finding for social science work that utilizes identifiers without accounting for the bias this can induce. In addition, we describe how CurmElo can be used to produce rankings of arbitrary features or dimensions of preference of a set of objects relative to a population of raters.