The Viúva Negra crawler: an experience report | Zendy

Gomes Daniel | Zendy; Silva Mário J. | Zendy

AI Assistant Blog Pricing

Home ZAIA Blog

Premium

The Viúva Negra crawler: an experience report

Author(s) -

Gomes Daniel,

Silva Mário J.

Publication year - 2008

Publication title -

software: practice and experience

Language(s) - English

Resource type - Journals

SCImago Journal Rank - 0.437

H-Index - 70

eISSN - 1097-024X

pISSN - 0038-0644

DOI - 10.1002/spe.825

Subject(s) - web crawler , focused crawler , crawling , world wide web , computer science , web page , web development , static web page , biology , anatomy

This paper documents hazardous situations on the Web that crawlers must address. This knowledge was accumulated while developing and operating the Viúva Negra (VN) crawler to feed a search engine and a Web archive for the Portuguese Web for four years. The design, implementation and evaluation of the VN crawler are also presented as a case study of a Web crawler design. The case study tested provides crawling techniques that may be useful for the further development of crawlers. Copyright © 2007 John Wiley & Sons, Ltd.

This content is not available in your region!

Continue researching here.

Having issues? You can contact us here

Accelerating Research