Visualizing meta search results: Evaluating the metacrystal toolset

MetaCrystal aims to help users find the documents that are most likely to be relevant. It compares the results returned by multiple search engines and displays the documents based on the number of engines that found them and their rank positions in the result lists. First, this paper examines the validity of the MetaCrystal design approach by calculating the probability that a document is relevant based on the number of engines that found it and its rank positions in the result lists. Using TREC data, it is shown that MetaCrystal's visual design is sound because it visually emphasizes documents based on their likelihood of being relevant. Second, this paper investigates if users are able to use the MetaCrystal's visualizations to find the documents that are most likely to be relevant. The results of a user study are reported that show that novice users are able to accomplish this task.