Do AI crawlers execute JavaScript? What the evidence says (2026)

Do AI crawlers execute JavaScript? For almost every bot that matters, no. According to Vercel’s December 2024 crawler study, the most recent published test of this question, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent, and Bytespider all fetch raw HTML and never run your client-side code. The two exceptions are Google’s Gemini, which rides Googlebot’s rendering infrastructure, and AppleBot, which crawls with a browser-based crawler. The answer holds however you phrase the question: no matter whether AI crawlers can execute JavaScript in theory, in practice the major ones simply don’t.

The short answer: most AI crawlers do not execute JavaScript

If your content exists only after JavaScript runs in a browser, ChatGPT, Claude, and Perplexity are working from a page they have never actually seen. Here is the capability matrix, crawler by crawler:

Two caveats worth stating plainly. First, this data is dated: the Vercel test ran in late 2024, and as of mid-2026 no AI vendor has publicly documented adding JavaScript rendering to its crawler. The picture could change with one announcement, which is exactly why every claim in this article carries a date. Second, we’re summarizing published third-party tests and official vendor documentation here, not a study of our own.

What the published evidence shows

The Vercel study is the spine of everything currently written on this topic, so it’s worth knowing what it actually measured. Vercel analyzed crawler traffic across its own network, including nextjs.org, and found that GPTBot made 569 million fetches and Anthropic’s Claude crawler 370 million in a single month, together roughly 20% of Googlebot’s 4.5 billion requests over the same period. AI crawling is no longer a rounding error next to search crawling.

The study’s most useful finding is a subtle one. AI crawlers download JavaScript files without running them: 11.50% of ChatGPT’s crawler fetches and 23.84% of Claude’s were JavaScript files. The bots can read that code as text, which may help them interpret a page’s structure, but they never execute it. A React component fetched as a .js file is source code to them, not rendered content.

They are also inefficient. Vercel measured that 34.82% of ChatGPT’s fetches and 34.16% of Claude’s returned 404 errors. Roughly a third of AI crawler requests hit dead URLs, mostly stale links and outdated asset paths. That has a practical implication we’ll come back to: broken URLs and stale sitemaps waste the limited attention AI crawlers give your site.

Scale-wise, the trend has only accelerated since. Cloudflare’s Radar data shows GPTBot’s share of AI-only crawler traffic grew from 5% in May 2024 to 30% in May 2025, a 305% increase in requests, making it the top AI crawler ahead of ClaudeBot at 21% and Meta-ExternalAgent at 19%. More AI crawler traffic, same rendering limitation: the gap between what browsers show and what AI sees is getting more expensive every month.

Crawler by crawler: what the official docs say

Third-party tests are half the evidence. The other half is what the vendors themselves document, and just as importantly, what they don’t. None of the official crawler docs from OpenAI, Anthropic, or Perplexity claims JavaScript rendering capability. For companies that document user-agent strings, IP ranges, and robots.txt behavior in detail, that silence is informative.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI documents three main crawlers, each with a separate robots.txt decision attached. GPTBot collects training data for future models. OAI-SearchBot builds the index behind ChatGPT search, and OpenAI explicitly recommends allowing it so your site can appear in ChatGPT search results. ChatGPT-User fires when a user asks ChatGPT to visit a page, and robots.txt rules may not apply to it because the visit is user-initiated.

If you’ve searched for “GPTBot JavaScript” hoping for an official statement, there isn’t one. Nothing in OpenAI’s documentation mentions rendering, and the Vercel test confirmed GPTBot reads initial HTML only.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic documents three user agents: ClaudeBot for training data collection, Claude-SearchBot for search quality, and Claude-User for user-initiated visits. Each can be blocked individually via robots.txt, and Anthropic supports the non-standard Crawl-delay directive, which is rare among AI vendors. As with OpenAI, the docs are silent on JavaScript execution, and the published testing shows ClaudeBot fetching HTML only.

Perplexity: PerplexityBot, Perplexity-User

Per the published crawler documentation, Perplexity draws a line the other vendors blur: PerplexityBot indexes pages to surface them in search results and is not used for training foundation models. Perplexity-User handles user-triggered visits and generally ignores robots.txt rules because a human asked for the page. Again, no rendering claim anywhere in the documentation.

Why Google renders JavaScript and AI crawlers don’t

Google is the outlier because it spent two decades building rendering infrastructure. Per Google Search Central’s JavaScript SEO documentation, Google processes JavaScript in three phases: crawling, rendering, and indexing. Pages get fetched, then queued for rendering in an evergreen version of headless Chromium, where they may wait seconds or longer before JavaScript actually executes. Even for Google, rendering is a deferred, resource-rationed step, not a free one.

That cost is the whole explanation for the gap. Executing JavaScript means running a full browser environment per page: fetching every script, executing it, waiting for network calls to settle. Multiply that across billions of pages and you get an infrastructure bill only a search incumbent has historically been willing to pay. The AI vendors, racing to ingest as much text as possible as cheaply as possible, skipped it. Their crawlers are closer to curl than to Chrome.

Here’s the nuance most write-ups on this topic bury: Google’s AI features inherit Googlebot’s rendering. Per Google’s AI features documentation, AI Overviews and AI Mode are fed by standard Googlebot crawling. There are no additional requirements, special files, or markup needed to appear in them, and standard controls like nosnippet and noindex apply. So a client-rendered React page that Google indexes fine can be cited in an AI Overview while remaining invisible to ChatGPT and Perplexity. We cover the Google side of this in detail in our guide to ranking in AI Overviews.

One related control worth knowing: Google-Extended is a robots.txt token that governs whether your content trains future Gemini models and grounds Gemini Apps and Vertex AI. Blocking it does not affect your inclusion or ranking in Google Search. It’s a training opt-out, not a search opt-out.

What this means if your site runs on React, Vue, or any SPA

Picture what a non-rendering bot receives when it fetches a client-side-rendered page: a near-empty HTML shell, a root div, and a stack of script tags. No product descriptions. No prices. No docs content. Everything your framework injects after load simply doesn’t exist for that visitor.

The practical damage concentrates in two site types. Ecommerce stores with client-rendered product data are invisible when a shopper asks ChatGPT to compare products in their category, even if those same pages rank fine in Google. SaaS documentation rendered client-side can’t be quoted when a developer asks an AI assistant how to integrate a tool, which is exactly the moment a citation would have driven a signup. Ranking in Google while being absent from AI answers is the default state of an unfixed SPA. We break down the underlying mechanics in our JavaScript SEO guide, and the SPA-specific version in React SEO without SSR.

Structured data follows the same rule with one twist. JSON-LD that ships in your initial HTML is readable by AI crawlers because it’s just text in the response body. JSON-LD injected client-side after load is not. Plenty of sites run tag managers or React helmet-style libraries that inject schema in the browser, pass Google’s Rich Results Test (which renders), and assume they’re covered. For non-rendering AI crawlers, that markup was never there.

How to check what AI crawlers actually see

You don’t need to guess any of this. Fifteen minutes with a terminal settles it for your own site.

1. Fetch your page the way the bots do. Request a JavaScript-dependent page with an AI crawler’s user-agent string (each vendor’s docs list the full string) and look at what comes back:

curl -sL -A "GPTBot" https://www.yourdomain.com/pricing/ | grep -i "your-key-product-claim"

If the phrase your buyers need to see isn’t in that response, ChatGPT has never read it.

2. Compare initial HTML with the rendered DOM. Open the page in a browser and diff “view source” (the raw response) against the DOM in devtools (post-JavaScript). Everything that exists only in the second view is invisible to non-rendering crawlers. Content present in both is safe.

3. Read your server logs. Grep access logs for GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot to see what these bots request and what status codes they get:

grep -E "GPTBot|ClaudeBot|OAI-SearchBot|PerplexityBot" access.log | awk '$9 == 404' | head

Remember the Vercel finding that roughly a third of AI crawler fetches end in 404s. If your logs show the same pattern, stale sitemaps and broken internal links are burning the crawl attention these bots allocate to you, on URLs that return nothing.

4. Check robots.txt for accidental blocks. Legacy blanket rules, or a 2023-era GPTBot block someone added and forgot, may be excluding the search bots you now want citations from.

This check is a standard part of a professional technical audit. If you want the full methodology, including rendering diffs at scale rather than page by page, our JavaScript SEO audit walkthrough covers the complete process.

How to make JavaScript content visible to AI crawlers

The fixes, in order of impact:

1. Server-side rendering or static generation for anything you want cited. Next.js, Nuxt, and Astro all ship complete HTML in the initial response, which is the entire requirement. This is the durable fix: it works for every crawler that exists today and every one that launches tomorrow.
2. Prerendering as a retrofit. If a rebuild isn’t realistic this quarter, a prerendering layer that serves pre-built HTML snapshots to bots closes the gap without touching your application code. It adds an infrastructure dependency and needs monitoring, but it beats invisibility.
3. Progressive enhancement for core content. Ship the substance (copy, prices, specs, answers) in initial HTML and let JavaScript enhance the experience on top. Interactive widgets can stay client-side; the content underneath them shouldn’t.
4. Server-rendered JSON-LD and clean semantic markup. Structured data belongs in the initial response, not in a client-side injection. Headings, tables, and lists in real HTML give a text-only crawler the document structure it needs to extract and quote you accurately.
5. Keep robots.txt open for the answer-engine bots you want citations from, OAI-SearchBot and PerplexityBot in particular.

There’s a business framing that makes prioritization easy: AI citations are a lead channel now. When a buyer asks ChatGPT to recommend vendors and the answer is assembled from initial-HTML content only, client-rendered competitors aren’t in the running, and neither are you if your stack ships empty shells. Invisible content generates zero leads. If you’d rather hand this off, our JavaScript SEO service handles the diagnosis and the rendering fix as a done-for-you engagement.

Should you block or allow AI crawlers?

Blocking is not one decision. Training bots and answer-engine bots do different jobs, and robots.txt lets you treat them separately.

Blocking GPTBot or ClaudeBot keeps your content out of future training runs and costs you nothing in AI search visibility. Blocking OAI-SearchBot or PerplexityBot removes you from ChatGPT search and Perplexity answers, the places where buyers increasingly do their vendor research. Those are opposite trades, and lumping them into one wildcard rule is how sites disappear from AI answers by accident.

Most sites haven’t made either call deliberately. Cloudflare found that only about 14% of the top-10,000 domains that had a robots.txt file (546 in all) carried any AI-bot directive as of June 2025, with GPTBot the most frequently blocked. The overwhelming default is no policy at all.

For a lead-generation site, our position is straightforward: allow the answer-engine bots, because citations there are demand you’d otherwise never see. Decide on the training bots separately, based on whether your content is a moat worth defending or marketing that works harder the more machines have read it. For most B2B sites, it’s the latter.

Where to go from here

The evidence points one way: ship your content in initial HTML, keep the answer-engine bots unblocked, and re-check the data every few months because this landscape moves fast. Run the curl test on your three most important pages this week; it will tell you in minutes whether you have a problem worth fixing.

Rendering is the entry ticket, not the whole game. Once AI crawlers can read your pages, the next question is whether they cite you over competitors, which comes down to how quotable, structured, and authoritative your content is. That’s the discipline our generative engine optimization service covers end to end, from crawlability through citation tracking. Fix visibility first; nothing downstream works without it.