Crawling is the way search engines find pages. A program called a crawler, spider or bot requests a web address, downloads what comes back, pulls out every link on it, and adds those links to a list of addresses to visit later. Repeat that billions of times and you have a map of the public web.
How a crawler moves through a site
A crawler starts from URLs it already knows: pages seen before, links from other sites, and URLs listed in XML sitemaps. Before fetching anything on a host it reads that host's robots.txt file to see which paths it may request. For each page it gets an HTTP status code: 200 means content, 301 or 302 means follow the redirect, 404 means gone, and 5xx errors tell it the server is struggling, which makes it slow down.
Links are the main road. A crawler only follows real <a href> links, so a menu built from buttons that change the page with JavaScript, without proper links, can leave whole sections undiscovered. Google's crawler, Googlebot, also renders JavaScript, but it does so in a later pass.
Crawling in practice: a new service page
An electrician adds a page about EV charger installation. If the page is linked from the homepage and the services menu, Googlebot will likely find it the next time it revisits those pages. If it is only reachable through a link in an email newsletter, Google may not find it for a long time. Adding it to the sitemap and linking it from a related page fixes that.
Does a small business need to think about this?
Mostly no. A small site with normal navigation gets crawled without help. Crawling becomes worth checking when new pages never appear in Search Console, after a platform migration, or if server errors spike. Search Console's Crawl Stats report shows how often Google visits and what responses it gets.
Things that quietly stop crawlers
- A leftover
Disallow: /in robots.txt from a test site, which blocks everything. - Links hidden behind forms or logins. Crawlers do not fill in search boxes or sign in.
- Redirect chains and loops that waste fetches and can end in an error.
- Pages linked from nowhere. These are orphan pages and may never be found.
Related terms
- Indexing: what happens to a page after it is crawled.
- Crawl budget: how much crawling a site gets.
- Internal linking: the paths crawlers follow inside your site.
- Googlebot: the best-known crawler.