Googlebot is the name of Google's search crawler, the automated visitor that requests pages from your site so Google Search can include them. When people say "Google hasn't seen my page yet", they mean Googlebot has not fetched it.
How Googlebot fetches and renders a page
There are two main versions: Googlebot Smartphone and Googlebot Desktop. Since Google moved to mobile-first indexing, the smartphone crawler does most of the work, so the mobile version of your page is the one Google judges. Googlebot first downloads the raw HTML, then queues the page for rendering in an up-to-date Chromium engine that runs your JavaScript. Content that only appears after scripts run is seen, but later than content in the initial HTML.
It obeys rules in your robots.txt file addressed to Googlebot or to all agents. It ignores the crawl-delay directive; if it is hitting your server too hard, slow responses or 503/429 status codes make it back off. Google also runs separate crawlers for other jobs, such as Googlebot-Image, and a robots token called Google-Extended that controls use of content for its AI models without affecting Search.
Spotting fake Googlebots
A bookshop owner sees thousands of requests in her server logs from a user agent claiming to be Googlebot. Anyone can type that string, so the label proves nothing. Google's documented check is a reverse DNS lookup on the IP address: a real Googlebot resolves to a hostname ending in googlebot.com or google.com, and a forward lookup on that hostname returns the same IP. Google also publishes its crawler IP ranges as JSON files. If the check fails, it is a scraper and can be blocked safely.
When a small business needs to care
Usually you never interact with Googlebot directly. It matters when you block it by accident, for example through a security plugin or firewall rule that treats bots as attacks, or when key text is loaded by scripts that break. Search Console's URL Inspection tool shows the page as Googlebot rendered it, which is the fastest way to confirm it sees what visitors see.
Common Googlebot mistakes
- Blocking CSS and JavaScript files in robots.txt, so Googlebot renders a broken page.
- Serving Googlebot different content from visitors. Showing the crawler one page and people another is treated as cloaking under Google's spam rules.
- Trusting the user-agent string when deciding to allow or block traffic.
- Testing only the desktop layout when the smartphone crawler is the one indexing you.
Related terms
- Crawling: the general process Googlebot carries out.
- Crawl budget: how many fetches Googlebot gives your site.
- Indexing: what Google does with the pages Googlebot brings back.
- Google Search Console: where you see Googlebot's activity on your site.