| |
Crawler
A crawler, sometimes called a robot, bot or spider is a program that combs the Web’s hypertext structure and retrieves and indexes the information it finds on Web sites.
A crawler retrieves documents it finds and recursively retrieves all the documents that are linked within them.
|
|
