Premium
Evaluating the retrieval effectiveness of web search engines using a representative query sample
Author(s) -
Lewandowski Dirk
Publication year - 2015
Publication title -
journal of the association for information science and technology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.903
H-Index - 145
eISSN - 2330-1643
pISSN - 2330-1635
DOI - 10.1002/asi.23304
Subject(s) - computer science , information retrieval , search engine , relevance (law) , web search query , sample (material) , web query classification , query expansion , metasearch engine , crowdsourcing , world wide web , chemistry , chromatography , political science , law
Search engine retrieval effectiveness studies are usually small scale, using only limited query samples. Furthermore, queries are selected by the researchers. We address these issues by taking a random representative sample of 1,000 informational and 1,000 navigational queries from a major G erman search engine and comparing G oogle's and B ing's results based on this sample. Jurors were found through crowdsourcing, and data were collected using specialized software, the Relevance Assessment Tool ( RAT ). We found that although G oogle outperforms B ing in both query types, the difference in the performance for informational queries was rather low. However, for navigational queries, G oogle found the correct answer in 95.3% of cases, whereas B ing only found the correct answer 76.6% of the time. We conclude that search engine performance on navigational queries is of great importance, because users in this case can clearly identify queries that have returned correct results. So, performance on this query type may contribute to explaining user satisfaction with search engines.