Share of answer is the operating metric for this practice.

How we count it
Request an audit

Insight · 11 min

AI crawlers read your HTML, not your app

Most AI crawlers do not run JavaScript. As of November 2026, the published evidence shows OpenAI’s, Anthropic’s, Perplexity’s, Meta’s, and ByteDance’s crawlers reading only the HTML your server returns, while Googlebot and Applebot render pages in a browser. If your text, links, or canonical tag appear only after a script runs, ChatGPT, Claude, and Perplexity see an empty shell, even when Google ranks the same page.

In short

  • Googlebot renders JavaScript, so Google Search, AI Overviews, and AI Mode can see client-rendered content. Most other AI crawlers cannot.
  • The main evidence is a December 2024 Vercel and MERJ log study. It found that OpenAI’s and Anthropic’s crawlers download JavaScript files but do not execute them.
  • A January 2026 test by RESONEO found that the user-triggered fetchers behind ChatGPT, Claude, and Gemini read raw HTML only. Copilot’s fetcher ran inline JavaScript but gave up on delayed content.
  • One site owner reported in September 2026 that OpenAI’s crawlers began requesting files that only exist after rendering. OpenAI has not confirmed a change. Plan for raw HTML.
  • The test is cheap: request the page without JavaScript and count the words. If the content is not in that response, fix it with server-side rendering, static generation, or prerendering.

Do AI crawlers render JavaScript?

Most do not. A crawler that does not render reads the HTML in the server’s response and stops. Anything a script adds afterwards, such as text from an API call, a product grid, or a navigation menu, never reaches it.

The best observational evidence is still Vercel’s study with MERJ, published December 17, 2024. It analyzed crawler traffic on nextjs.org and the Vercel network and states that “none of the major AI crawlers currently render JavaScript.” The list it gives is OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider), and Perplexity (PerplexityBot). The same study found that JavaScript files made up 11.50% of ChatGPT’s crawler requests and 23.84% of Claude’s. Both bots fetch the files and do not execute them. A JavaScript request in your logs is therefore not evidence of rendering.

Two crawlers in that study did render. Gemini uses Googlebot’s infrastructure, and Applebot renders through a browser-based crawler. Apple’s own Applebot documentation agrees: it warns that if JavaScript, CSS, or other resources are blocked, Applebot “may not be able to render the content properly.”

None of the AI operators documents its rendering behavior. OpenAI’s crawler overview says what each bot is for and how robots.txt applies, and nothing about JavaScript. Everything below about rendering comes from observation, not from vendor statements.

Which bots render, and what is the evidence?

The table below is the state of the public record as of November 2026. “No” means a published test or log study observed no JavaScript execution. It does not mean the operator has ruled it out.

BotOperatorFeedsRuns JavaScript?Evidence
GooglebotGoogleGoogle Search, AI Overviews, AI Mode, Gemini groundingYes, in a deferred render queueGoogle Search Central; Vercel/MERJ 2024
ApplebotAppleSpotlight, Siri, Safari searchYes, browser-basedApple documentation; Vercel/MERJ 2024
BingbotMicrosoftBing, Copilot groundingGenerally, but not on every page at scaleBing Webmaster blog, 2018
GPTBot, OAI-SearchBotOpenAITraining; ChatGPT searchNo. Unconfirmed signal of change in September 2026Vercel/MERJ 2024; one site owner’s logs, 2026
ChatGPT-UserOpenAILive fetches in ChatGPTNo, raw HTML onlyVercel/MERJ 2024; RESONEO 2026
ClaudeBotAnthropicTrainingNoVercel/MERJ 2024
Claude-UserAnthropicLive fetches in ClaudeNo, raw HTML onlyRESONEO 2026
Claude-SearchBotAnthropicClaude search indexNo published test foundNone
PerplexityBotPerplexityPerplexity searchNoVercel/MERJ 2024
Perplexity-UserPerplexityLive fetches in PerplexityNot determinedRESONEO 2026
Gemini live fetchGoogleGemini app fetching a URLNo, raw HTML onlyRESONEO 2026
Copilot live fetchMicrosoftCopilot fetching a URLInline JavaScript yes; delayed content missedRESONEO 2026
Meta-ExternalAgentMetaTraining, indexingNoVercel/MERJ 2024
Bytespider, CCBotByteDance, Common CrawlTraining, open archiveNoVercel/MERJ 2024

Three notes on the table.

The Gemini rows differ on purpose. Gemini answers grounded in Google’s index inherit Googlebot’s rendering. When the Gemini app fetches a URL live, RESONEO’s test found it read only the raw HTML. The same brand can see two different versions of your page.

RESONEO’s test covers user-triggered fetchers only. It planted 15 sentinel strings on a page using different injection methods, from static HTML to Shadow DOM, iframes, delayed scripts, and fetch() calls, then asked each assistant to report what it found. Copilot saw inline JavaScript but missed delayed JavaScript, with a timeout the study measured at about 53 milliseconds. Perplexity’s fetcher sent empty headers, so no result could be read. RESONEO is a vendor and does not state how many runs it made. Treat it as a well-designed probe, not an audit.

Bing’s statement is old. The October 2018 bingbot post says “bingbot is generally able to render JavaScript” and that “it is difficult for bingbot to process JavaScript at scale on every page of every website.” That matters for Copilot, because Microsoft’s web query documentation says Microsoft 365 Copilot grounds answers by sending search queries to Bing. Partial rendering at scale is not something to rely on.

Why can Google’s AI Overviews see a page that ChatGPT cannot?

Because AI Overviews read Google’s index, and Google’s index is built by a crawler that renders. Google states that the only technical requirement for AI Overviews and AI Mode is that a page “must be indexed and eligible to be shown in Google Search with a snippet,” with “no additional technical requirements” (Google Search Central). How AI Overviews choose a source covers what happens after the page is indexed.

Google processes JavaScript in three phases: crawl, render, index. Its JavaScript SEO basics say a page “may stay on this queue for a few seconds, but it can take longer than that.” So a client-rendered page can rank in Google, appear in an overview, and still be invisible to the bots behind ChatGPT search, Claude, and Perplexity. Teams that check only Google conclude rendering is fine. It is fine for one engine.

This is the retrieval layer that SEO, AEO, and GEO all sit on. A page a crawler cannot read cannot be extracted into an answer or cited by a generative engine. Rendering is a technical SEO problem with GEO consequences.

What does a non-rendering crawler miss?

It misses everything a script adds after the first response. On a client-rendered single-page app, that is often the whole page. An illustrative response from such an app looks like this:

<!doctype html>
<html>
  <head>
    <title>Acme</title>
    <meta name="description" content="Web site created using create-react-app">
  </head>
  <body>
    <div id="root"></div>
    <script src="/static/js/main.4f2a1c.js"></script>
  </body>
</html>

A browser runs the script and shows a product page. A non-rendering crawler gets a title, a default description, and an empty div. That is not a hypothetical shape. An October 2026 audit by aat.ee requested 342 recently launched product sites with an OAI-SearchBot user agent and found that 37 (10.8%) returned fewer than 50 words of body text, and 30 of those 37 used an empty root or app div. Only 1 of 165 Next.js sites was thin. The audit simulated the crawler from its own network and checked one URL per site, so it measures what the raw HTML contains, not what OpenAI actually indexed.

The loss is wider than body text. Check each of these:

ElementIf it is added by JavaScript
Body text and headingsNot read. Nothing to extract or cite
Internal linksNot followed. Google can only discover links that are <a> elements with an href attribute, and other crawlers are no better
Canonical tagNot seen by non-rendering bots. Google says not to use JavaScript to change the canonical to a different URL than the HTML states
Meta robots noindexGoogle “may skip rendering and JavaScript execution” when it sees noindex, so removing it with script may not work
Structured dataJSON-LD injected by a tag manager is invisible to non-rendering bots
Status codesA single-page app that shows “not found” with a 200 status creates soft 404s

The structured-data row matters if you added schema for AI search. Markup that exists only after a tag manager fires does not reach most AI crawlers, whatever schema does for AI search once it is read.

Have OpenAI’s crawlers started rendering JavaScript?

Possibly on some sites, and it is not confirmed. On September 25, 2026, one developer saw GPTBot and OAI-SearchBot start requesting Next.js ?_rsc= prefetch URLs, which a client only generates after rendering. The pattern appeared on four of the author’s Next.js sites, and the requests were checked against OpenAI’s published IP ranges (Digital Drift).

The author’s own caveats are the right reading. It is one person’s sites. OpenAI made no announcement. Cloudflare’s aggregate OpenAI traffic stayed flat through September, which suggests a limited rollout at most. Most rendered pages had not been freshly fetched that day, so OpenAI may be rendering stored HTML in a separate process rather than at crawl time.

The decision rule does not change. Rendering, if it arrives, will be partial, delayed, and undocumented, as Google’s queue still is. Content you want cited should be in the HTML response. Revisit this when an operator documents its renderer.

How do you check what an AI crawler sees?

Request the page the way a non-rendering bot does, then compare it with what a browser shows. Four checks cover most sites.

  1. Fetch the raw HTML with the bot’s user agent. This also catches CDN or firewall rules that treat bots differently. The aat.ee audit found three sites that returned 403 to the crawler string and 200 to Chrome.
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot" \
  -o bot.html -w "%{http_code} %{size_download}\n" https://www.example.com/page
  1. Search the response for a sentence you want quoted. Pick one from the middle of the page, not the title. If grep cannot find it in bot.html, no non-rendering crawler can either.
  2. Count words without scripts. Strip <script> and <style> blocks and tags, then count. Compare with the same count on the rendered page from your browser’s developer tools. A large gap is client-side rendering.
  3. Check Google separately. Search Console’s URL Inspection tool shows the HTML Googlebot received and the rendered result. A pass there tells you about Google only.

Turning JavaScript off in a browser is a fast visual check of the same thing. For whether the bots are arriving at all and getting 200s, use the log checks in robots.txt for AI crawlers. A spoofed user agent from your own network shows what your server returns. It does not show what the operator stored.

What should you change?

Put the content you want retrieved into the server’s HTML response. Google’s guidance lists the options: “use server-side rendering, static rendering, or hydration as a solution” (dynamic rendering). Google also notes that “not all bots can run JavaScript,” which is the whole problem in six words.

ApproachWhat the bot receivesWhen it fits
Static generationComplete HTML built at deploy timeMarketing pages, docs, articles, anything that changes per release
Server-side rendering with hydrationComplete HTML per request, made interactive in the browserPages with frequently changing data, such as pricing or inventory
Prerendering serviceA cached HTML snapshot served to botsA legacy single-page app you cannot rebuild soon
Dynamic renderingDifferent HTML for bots than for usersNot recommended. Google calls it “a workaround and not a long-term solution”

Bing recommended dynamic rendering in 2018. Google now calls it a workaround. A prerendering layer keyed on user agents is a stopgap. It has to know every bot token, and it fails silently when a new one appears.

Not everything needs to be in the HTML. Vercel’s study suggests keeping client-side rendering for non-essential features. View counters, chat widgets, and interactive calculators can load later. Product descriptions, prices, specifications, comparison tables, the claims you want quoted, primary navigation links, the canonical tag, and structured data cannot.

Sequence the work by the queries it unblocks. Start with the pages that carry the claims you want cited, which is what what gets cited argues for, not with the whole template library.

Worked example: what an AI crawler gets from this site

AIEOLABS is built with Astro and deployed as static HTML. On November 7, 2026, we requested our own llms.txt guide twice from our own network: once with the OAI-SearchBot user agent shown above, once with a Chrome user agent.

  • Both requests returned 200 and the same 50,234 bytes.
  • After stripping script and style blocks, both responses contained 2,394 words.
  • The only <script> elements were two application/ld+json blocks. There is no executable JavaScript on the page, so there is nothing to render.
  • The canonical tag and all 50 links were in the raw response.

This is a test of what our server returns to that user agent. It is not a test of what OpenAI fetched or indexed, because the requests came from our IP, not OpenAI’s. For a site that renders on the server, that is the question that matters. For a client-rendered site, the same two requests would show the gap in one line of output.

Common mistakes

  • Testing only in Google. URL Inspection shows rendered HTML because Googlebot renders. Test the raw response too.
  • Reading JavaScript requests in logs as rendering. OpenAI’s and Anthropic’s crawlers fetch script files without running them.
  • Injecting the canonical or noindex with script. Put both in the server HTML, and never use script to change them.
  • Adding JSON-LD through a tag manager. Render structured data on the server if you want non-Google bots to read it.
  • Navigation built from onclick handlers. Use <a href> links, so every crawler can follow them.
  • Blocking JavaScript and CSS in robots.txt. It can stop Googlebot and Applebot from rendering. Allow the resources a page needs.
  • Prerendering for a hard-coded bot list. Operators keep adding tokens. Serve the same complete HTML to everyone.
  • Fixing every template at once. Fix the pages you want cited first, then measure whether citations change.

Sources

Bring us the domain.

We will tell you where you are absent, which page is in the way, and what has to be true before a citation is possible.

Request an audit