Towards a highly-scalable and effective metasearch engine

A metasearch engine is a system that supports uni ed access to multiple local search engines. Database selection is one of the main challenges in building a large-scale metasearch engine. The problem is to eAEciently and accurately determine a small number of potentially useful local search engines to invoke for each user query. In order to enable accurate selection, metadata that re ect the contents of each search engine need to be collected and used. In this paper, we propose a highly scalable and accurate database selection method. This method has several novel features. First, the metadata for representing the contents of all search engines are organized into a single integrated representative. Such a representative yields both computation eAEciency and storage eAEciency. Second, our selection method is based on a theory for ranking search engines optimally. Experimental results indicate that this new method is very e ective. An operational prototype system has been built based on the proposed approach.