Premium
Effective page refresh policy
Author(s) -
Gao Kai
Publication year - 2007
Publication title -
computer applications in engineering education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.478
H-Index - 29
eISSN - 1099-0542
pISSN - 1061-3773
DOI - 10.1002/cae.20155
Subject(s) - computer science , web crawler , relevance (law) , process (computing) , resource (disambiguation) , search engine , web page , interval (graph theory) , bandwidth (computing) , world wide web , information retrieval , data mining , operating system , computer network , mathematics , combinatorics , political science , law
Web pages are created or updated randomly. As for a search engine, keeping up with the evolving Web is necessary. But previous studies have shown the crawler's refresh ability is limited because it is not easy to detect the change instantly, especially when the resources are limited. This article concerns modeling on an effective Web page refresh policy and finding the refresh interval with minimum total waiting time. The major concern is how to model the change and which part should be updated more often. Toward this goal, the Poisson process is used to model the process. Further, the relevance is also used to adjust the process, and the probability on some sites is higher than others so these sites will be given more opportunities to be updated. It is essential when the bandwidth is not wide enough or the resource is limited. The experimental results validate the feasibility of the approach. On the basis of the above works, an educational search engine has been developed. © 2007 Wiley Periodicals, Inc. Comput Appl Eng Educ 14: 240–247, 2007; Published online in Wiley InterScience (www.interscience.wiley.com); DOI 10.1002/cae.20155