What Is a Web Crawler?
How Web Crawlers Discover Pages
A web crawler is an automated program that visits web pages, reads their content and follows the links it finds to reach more pages. Search engines use crawlers to collect the material they later index and rank.
Crawlers start from a list of known URLs. Each time they fetch a page, they note every link on it and add any new addresses to their queue. Sitemaps supplied by site owners are another source of URLs, which is why a page that no other page links to can still be discovered. Crawlers also revisit known URLs to check for changes, so a page that is updated often is usually fetched more frequently than one that rarely changes.
Googlebot, Bingbot and Other Search Crawlers
Each search engine runs its own crawlers. Googlebot is the name of Google's main crawler, and Bingbot is Microsoft's equivalent for Bing. Other crawlers serve specific purposes, such as fetching images, videos or news content.
Every crawler identifies itself with a user agent, a short text string sent with each request. Site owners can see these names in their server logs and can address specific bots in their robots.txt file. A user agent name alone is not proof of identity, because any program can copy it. That is why verification matters.
Verifying That a Visitor Is a Real Crawler
Many programs pretend to be Googlebot in order to bypass rules or copy content. Google recommends checking a visitor's IP address with a reverse DNS lookup, which should resolve to a hostname in Google's crawler domains. It then recommends a forward lookup on that hostname to confirm that it points back to the same IP address. Google also publishes the IP ranges its crawlers use.
Bingbot can be verified in a similar way using Microsoft's own guidance. Verification matters when you decide which traffic to block, limit or count in analytics.
Crawl Rate and Server Load
The crawl rate is how many requests a crawler makes to a site over a given period. Search engines try not to overload servers. If a site responds slowly or returns many errors, the crawler slows down. If it responds quickly and reliably, the crawler may increase its requests, within the limits of how much the content is worth fetching. The wider limit on crawling is discussed in crawl budget.
Site owners influence this mainly through server capacity. Bing respects a non-standard crawl-delay directive in robots.txt, while Google does not honor it and adjusts its own rate automatically. Crawl activity can be reviewed in the Crawl stats report of Google Search Console, which shows requests, response times and response types.
Frequently asked questions
Can I Block Googlebot From My Site?
You can ask Googlebot not to crawl parts of a site with robots.txt, but blocking crawling is not the same as keeping a page out of search. Pages can still appear in results if other sites link to them. To keep a page out of results, use a noindex instruction that the crawler is able to read, as explained in robots.txt vs noindex.
How Often Does Googlebot Visit a Website?
There is no fixed schedule. Googlebot visits popular and frequently updated pages more often, and it visits slow sites or rarely changing sites less often. Server speed and errors also shape how many requests the crawler makes. Site owners can see actual visit patterns in their server logs or in crawl statistics.