Premium
How and why do users change their assessment of search results over time?
Author(s) -
ZhitomirskyGeffet Maayan,
BarIlan Judit,
Levene Mark
Publication year - 2015
Publication title -
proceedings of the association for information science and technology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.193
H-Index - 14
ISSN - 2373-9231
DOI - 10.1002/pra2.2015.145052010067
Subject(s) - relevance (law) , categorical variable , psychology , set (abstract data type) , rank (graph theory) , social psychology , phenomenon , period (music) , cognitive psychology , applied psychology , computer science , statistics , mathematics , political science , physics , combinatorics , acoustics , law , programming language , quantum mechanics
In this study we investigate whether and why users change their preferences when assessing search engine results over time. We conducted a study with 35 subjects who were asked to rank and assign relevance scores to the same set of search results for three times, with a few weeks period between each round. The subjects were then exposed to the differences in their judgements and were asked to explain them. A new coefficient to measure change was introduced to assess the results of the experiment. We found that all the subjects judge the vast majority of the results differently in every round. However, there was less change in relevance judgements than in rankings. Most of the subjects were satisfied with their changes, and did not perceive them as mistakes but rather as a legitimate phenomenon, since they believe that time influences the relevance assessment. Our analysis reveals that the main factors that caused these changes were due to categorical thinking, influence of the learnt information, and environmental and emotional changes.