Exploring evaluation criteria of social navigational tools on social media

A user study of aNobii was conducted to compare its three book-finding tools: author search, browsing friends' bookshelves and browsing similar bookshelves. The construct of \"social navigation\" was identified as a useful theoretical framework to discuss various modes of information access on social media. A within-subject experimental design was adopted where all forty regular aNobii users searched alternately with the three book-finding tools. Several novel evaluation measures were designed to explore the potential benefits these tools might bring to the users. Other than the self-report user experience and search result measures, the \"consideration set\" model was used as a novel framework for navigational effectiveness. Some major findings are as follows. While the author search function was shown to be the most efficient, browsing friends' bookshelves was shown to generate more interesting and informative browsing experience. Three evaluative dimensions were derived from our study: search experience, search efficiency, and search result quality. The disagreement of these measures shows a need for a multi-faceted evaluative framework for these exploration-based navigational tools.