by Steve Smith » Fri Jun 22, 2012 5:46 am
Googlebot is Google's web crawling bot. Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index. Googlebot's crawl process begins with a list of webpage URLs, generated from previous crawl processes and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links (SRC and HREF) on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.