Crawl budget is the amount of attention a search engine gives your site: roughly, how many of your URLs its crawler will fetch over a period of time. Google does not crawl every page every day, so on a big site the pages it spends that attention on are the ones that get updated in search first.
The two halves Google describes
Google's documentation splits crawl budget into two parts.
- Crawl capacity limit: how hard Googlebot can hit your server without hurting it. Fast, error-free responses raise it; slow responses and 5xx errors lower it.
- Crawl demand: how much Google wants to crawl. Popular pages, pages that change often and URLs it has never seen raise demand; stale or low-value pages lower it.
Your budget is whatever Google chooses within both limits. Every URL fetched counts, including redirects, duplicate pages, alternate versions and resource files.
Where budget gets wasted: an online store example
A clothing shop has 2,000 products, but its filter menu creates a new URL for every combination of size, colour, price and sort order. Googlebot finds hundreds of thousands of near-identical addresses like ?colour=red&size=m&sort=price and spends its visits there, while new product pages wait. Fixes include blocking the filter parameters in robots.txt, linking only to clean category URLs, and keeping the XML sitemap limited to real product and category pages.
Honestly, does your site need to worry?
Google says crawl budget is mostly a concern for very large sites (it mentions roughly a million or more unique pages) or medium sites of around ten thousand pages whose content changes daily. A typical small-business site with a few dozen pages can ignore it. If a small site has pages that are not indexed, the cause is almost always quality, duplication or a block, not budget.
Crawl budget mistakes
- Using noindex to save budget. Google still has to crawl a page to see the noindex tag.
- Long redirect chains that turn one visit into three or four fetches.
- Infinite spaces such as calendars with a "next month" link that never ends.
- A slow, overloaded server, which directly lowers the capacity limit.
- Blaming budget for thin content that Google simply does not want to index.
Related terms
- Crawling: the activity a crawl budget measures.
- Robots.txt: the main tool for keeping crawlers off wasteful URLs.
- Technical SEO: the wider discipline crawl budget belongs to.
- 302 redirect: one of the redirect types that adds extra fetches when chained.