Spiders (Web Crawlers, Searchbots)
Robot programs that are used by search engines to roam the Web and update the search engine database for deletions, additions, and updates of Web site information.
Even with many robots operating all the time, it can take Google, the largest search engine, a couple of months to completely comb the Web. Inevitably, some information is always obsolete.
There are times when you may want to block search engines from finding your Web site or specific pages on your site, such as for privacy reasons. Special HTML code or a special file is used to accomplish this.
See Link Rot.