Open AccessAn algorithm for labels aggregation in taxonomy-based crowd-labeling
Author(s) -
Andrew Ponomarev,
Tatiana Levashova,
N. Mustafin
Publication year - 2021
Publication title -
journal of physics. conference series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.21
H-Index - 85
eISSN - 1742-6596
pISSN - 1742-6588
DOI - 10.1088/1742-6596/1801/1/012012
Subject(s) - crowdsourcing , computer science , taxonomy (biology) , set (abstract data type) , software , information retrieval , data mining , machine learning , artificial intelligence , world wide web , botany , biology , programming language
Crowdsourcing provides a convenient solution for many information processing problems that are still hard or even intractable by modern AI techniques, but are relatively simple for many people. However, complete crowdsourcing solution cannot go by without a quality control mechanisms, as the results received from participants are not always reliable. The paper considers taxonomy-based crowd-labeling - a form of crowdsourcing, in which participants label objects with tags, and there exists an explicit taxonomy relation on the set of tags. We propose a method and an algorithm for label aggregation, allowing to estimate the likelihood of the true object label from a set of noisy labels received from the crowd, and to estimate the expected crowd members’ accuracy. The proposed method and algorithm can be used in a wide range of crowd-labeling applications (e.g., classification of scientific literature collections, software repositories, etc.).