Premium
Comparing response latency and self‐report methods for estimating levels of certainty in knowledge elicitation for rule‐based expert systems
Author(s) -
Feinstein Jerald L.
Publication year - 2000
Publication title -
expert systems
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.365
H-Index - 38
eISSN - 1468-0394
pISSN - 0266-4720
DOI - 10.1111/1468-0394.00144
Subject(s) - certainty , computer science , analytic hierarchy process , consistency (knowledge bases) , latency (audio) , feeling , confidence interval , artificial intelligence , machine learning , operations research , psychology , statistics , social psychology , mathematics , telecommunications , geometry
Many practitioners and theoreticians feel uncomfortable in using classic self‐report, knowledge acquisition methods for assessing an expert’s feelings of confidence or certainty about rules, choices or decisions. This paper proposes and tests an alternative approach using response latency, or the time it takes an expert to respond to a paired choice inquiry. The question is to learn if there is any merit in considering the response latency approach as a replacement for the classic method. This investigation is concerned with using the analytic hierarchy process (AHP) to compare two methods for estimating a subject’s feeling of certainty or confidence about their rule, choice or decision. One method is the older or classic method where a subject self‐reports both the choice selected and the level of certainty associated with a choice‐pair on the 1‐to‐9 scale used with the AHP. The second is proposed by the authors and based on using only the choice and decision time or response latency. In the second case, the inverse of response latency is used to estimate a subject’s degree of certainty or confidence in the selected choice in a paired comparison experiment. The response latency method has the advantage of being unobtrusive, less prone to conscious censure, quicker to perform, requiring less effort, and possibly being less expensive to administer. Employing AHP methodology, priorities are used to represent the degree of confidence among choices or rules, and consistency ratios are used to estimate the degree of consistency detected by each method. To compare the two methods, 21 subjects were evaluated in a paired sample design. Employing a computer‐administered questionnaire, subjects expressed relative levels of confidence for a set of potential recommendations and the time for each subject to respond was recorded unobtrusively. Using paired sample t tests, sufficient evidence was found to reject the null hypothesis (α= 0.025) that the mean of the difference in the priorities between the two methods was zero (p < 0.0001). In a second test, there was sufficient evidence to be at least 95% confident of accepting an alternative hypothesis that response latency produces a lower consistency ratio (higher consistency of the reports) than the self‐report method (p < 0.0001). In addition to the tests of significance, the results were found to be practically significant and are discussed in the paper.