Design of an Efficient Migrating Crawler based on Sitemaps

As the size of web keep on growing, the job of a web crawler has become more cumbersome in covering maximum web. There are enormous numbers of web pages available over the web but out of them a smaller number is available to user. A sitemap is an XML file in which a list of URLs of that particular site is mentioned. The sitemap protocol may play a very important role in covering maximum web. In this paper, the information provided by sitemap protocol is used for the purpose of crawling the quality web pages. With the help of sitemap, web crawlers will maintain their repository up-todate. It also tries to make user to access maximum pages over the web by covering as maximum as possible while crawling. It also helps crawler to visit the pages based on their change frequency and downloads updated pages only, thereby reduces unnecessary network traffic.