Premium
Inferring Expertise in Knowledge and Prediction Ranking Tasks
Author(s) -
Lee Michael D.,
Steyvers Mark,
de Young Mindy,
Miller Brent
Publication year - 2012
Publication title -
topics in cognitive science
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.191
H-Index - 56
eISSN - 1756-8765
pISSN - 1756-8757
DOI - 10.1111/j.1756-8765.2011.01175.x
Subject(s) - ranking (information retrieval) , computer science , set (abstract data type) , cognition , rank (graph theory) , measure (data warehouse) , machine learning , artificial intelligence , order (exchange) , cognitive psychology , data science , psychology , data mining , mathematics , finance , combinatorics , neuroscience , economics , programming language
We apply a cognitive modeling approach to the problem of measuring expertise on rank ordering problems. In these problems, people must order a set of items in terms of a given criterion (e.g., ordering American holidays through the calendar year). Using a cognitive model of behavior on this problem that allows for individual differences in knowledge, we are able to infer people's expertise directly from the rankings they provide. We show that our model‐based measure of expertise outperforms self‐report measures, taken both before and after completing the ordering of items, in terms of correlation with the actual accuracy of the answers. These results apply to six general knowledge tasks, like ordering American holidays, and two prediction tasks, involving sporting and television competitions. Based on these results, we discuss the potential and limitations of using cognitive models in assessing expertise.