JavaScript Websites and AI Crawlers: Can They See Your Pages? | How-to Guide

Why a site that looks fine in a browser can look empty to AI crawlers, what Google says about rendering JavaScript, how to test what bots receive, and the fixes that work for small business sites.

Many modern websites send the browser an almost empty HTML file and a bundle of JavaScript that builds the page on arrival. People never notice. Crawlers might. If a bot reads the HTML and does not run the scripts, your carefully written service page is, to that bot, a blank sheet with a logo.

How Google handles it

Google Search Central's JavaScript SEO basics explain that Googlebot crawls a page, queues it for rendering in a recent version of Chromium, and then indexes the rendered result. So Google can see JavaScript content, though rendering may happen later than the first crawl, and Google still recommends server-side rendering or pre-rendering because it is faster for users and for all crawlers. Its guidance also warns that links must be real anchor tags with an href to be followed.

How AI crawlers handle it

The AI companies say less about rendering. OpenAI documents OAI-SearchBot and GPTBot, Perplexity documents PerplexityBot, and none of those pages promises that the crawler executes JavaScript. Vercel's December 2024 analysis of crawler traffic on its network found that none of the major AI crawlers it measured rendered JavaScript, naming OpenAI's and Anthropic's crawlers and PerplexityBot. It found two exceptions: Gemini, which uses Googlebot's infrastructure, and Applebot. Crawlers change, so treat that as a snapshot, not a promise.

The safe assumption: if the words are not in the HTML your server sends, ChatGPT search, Perplexity and similar tools may not see them.

Test what a crawler receives

  1. Open your page and choose View Page Source (not Inspect, which shows the page after scripts run). Search for a sentence from your main heading. If it is there, the text is in the raw HTML.
  2. In Google Search Console, use URL Inspection and then View Crawled Page to see the HTML Google stored and a screenshot of the render.
  3. From a terminal, fetch the page with a plain request and read what comes back. If you see only a root div and script tags, crawlers that do not render get nothing.
  4. Turn off JavaScript in your browser's settings and reload. What remains is roughly what a non-rendering bot reads.

Fixes, from most to least robust

  • Server-side rendering or static generation. The server sends finished HTML. Frameworks such as Next.js, Nuxt and Astro support this, and many site builders produce static pages already.
  • Pre-rendering. A build step or service saves a full HTML snapshot of each page. Google treats this as acceptable as long as bots and people get the same content.
  • Put critical facts in the initial HTML. If a rebuild is not possible, at least render name, services, areas, hours and contact details on the server.

Avoid dynamic rendering that gives bots a different page from people. Google describes dynamic rendering as a workaround, not a long-term solution, and content that differs between bots and visitors crosses into cloaking.

Common traps on small business sites

  • Text placed inside images or sliders, which no crawler reads as text.
  • Menus built from buttons with click handlers instead of links, so inner pages are never discovered.
  • Reviews, prices or FAQs loaded from a third-party widget after the page loads.
  • A homepage that renders on the server while inner pages do not.

Check after every redesign

Rendering problems often arrive with a new theme or plugin. After any big change, repeat the page source check on the homepage and two inner pages.

We.Inc sites serve their inner pages as HTML that Google can read, and SEO God Mode builds on that by adding new, readable pages from your own services each week.

Frequently asked questions

Does Googlebot run JavaScript?

Yes. Google renders pages with a recent Chromium before indexing, though rendering can lag the first crawl. Google still recommends server-side rendering or pre-rendering where possible.

Do ChatGPT and Perplexity crawlers run JavaScript?

Their documentation does not say they do, and Vercel's 2024 crawler analysis found they did not render JavaScript. Put important text in the HTML your server sends.

How do I see what a crawler sees?

Use View Page Source in your browser, or URL Inspection and View Crawled Page in Google Search Console. If your words are missing there, non-rendering bots miss them too.

More in Guides

Website builders for the businesses in this article

Related guides and tools

Start building

Product

Who It's For

Features

Resources

Company

View Sitemap