Key fact: most AI crawlers don’t run JavaScript
Most engineers know this fact, but most bosses don’t — which is why there are so many stories of “we spent a fortune building a beautiful SPA, but AI citations are 0.”
The table below summarizes the JS execution capabilities of the major AI crawlers in 2026 (a combined observation from each vendor’s public docs + third-party testing; individual behavior may change over time):
| Crawler | Executes JavaScript? | Purpose |
|---|---|---|
GPTBot (OpenAI training) |
❌ No | Crawling training corpus |
ChatGPT-User (live citation) |
⚠️ Partial (simple JS) | Live citation in conversation |
ClaudeBot (Anthropic training) |
❌ No | Crawling training corpus |
PerplexityBot (Perplexity) |
⚠️ Partial | Search + live citation |
Google-Extended (Gemini training) |
⚠️ Partial (shares Googlebot) | Training corpus |
CCBot (Common Crawl) |
❌ No | Open dataset, used by many LLMs |
Googlebot |
✅ Yes (Chrome renderer) | Google search index |
Bingbot |
⚠️ Partial | Bing + ChatGPT browse |
Conclusions:
- Training-corpus crawlers (
GPTBot/ClaudeBot/CCBot) almost all don’t execute JS - Live-citation crawlers (
ChatGPT-User/PerplexityBot) partially execute simple JS, but complex React / Vue hydration mostly fails - Only Google’s own crawlers fully execute JS (which is exactly why the SEO crowd used to feel that “JS doesn’t affect search” — that’s only true for Google)
How to quickly verify which type your site is
Open a terminal and run:
# Simulate what GPTBot sees
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
-L https://yoursite.com/ | grep -c "some key content on your site"
Replace “some key content on your site” with a piece of text you expect to appear (e.g., your service name, a testimonial keyword).
Interpretation:
- Returns
1or more → AI can see it: SSR / SSG / Hybrid - Returns
0→ AI can’t see it: pure SPA or blocked - Returns
0but the browser shows it → the classic SPA shell problem
Advanced verification (check the full HTML size):
curl -s -A "GPTBot/1.0" https://yoursite.com/ | wc -c
Interpretation:
- < 5,000 bytes with a near-empty
<body>→ pure SPA - 5,000–50,000 bytes with substantive content → one of SSR / SSG / Hybrid
-
100,000 bytes → possibly SSR but with a lot of inline JS / CSS (the HTML-size dimension will deduct points)
The 4 rendering strategies in detail
Strategy 1: SSG (Static Site Generation) — highest citation rate
Examples: Hugo, Astro, Jekyll, 11ty, Docusaurus, Next.js export, Gatsby
How it works: At build time, every page is pre-rendered into HTML; at deploy time it’s pure static files.
For AI crawlers: Perfect. AI gets fully rendered HTML and doesn’t need to execute any JS.
Benefits:
- Each URL maps to one .html file
- CDN-cache friendly (any CDN can serve static files)
- Extremely fast load times
- No server compute cost
Limitations: - Content updates require a rebuild (deploy time ranges from minutes to hours) - Not suitable for apps with lots of dynamic content or real-time data
Good for: blogs, marketing sites, documentation sites, product info sites, landing pages
Strategy 2: SSR (Server-Side Rendering) — high citation rate
Examples: Next.js (getServerSideProps), Nuxt SSR, Rails + ERB, Django, Laravel + Blade
How it works: The server renders HTML on the fly for each HTTP request and returns it.
For AI crawlers: Perfect. Every request returns full HTML.
Benefits: - Real-time content, no build step needed - Can personalize per user (login state / region / locale) - Suitable for e-commerce, social, dashboards
Limitations: - Every request needs server compute (operational cost) - Slower TTFB (Time To First Byte) - CDN caching needs more careful design
Good for: e-commerce, membership apps, frequently updated content sites
Strategy 3: Hybrid (SSR + Client Hydration) — citation rate depends on implementation
Examples: Next.js (default), Nuxt, SvelteKit, Remix, Astro Islands
How it works: 1. First request: the server renders full HTML (AI can see it) 2. Subsequent interactions: client-side React/Vue/Svelte takes over, and JS renders the remaining content
For AI crawlers: depends on whether the key content is in the initial HTML
Benefits: - AI gets the first full HTML - User interaction experience is equivalent to a SPA - Balances SEO and UX
Pitfalls:
- If key content (product listings, article body, FAQ) is client-side fetch + render, AI still can’t see it
- A hydration mismatch can make the version AI sees inconsistent with what a real person sees
- M3-18 fixed exactly this kind of problem: geoweb.tw’s early <article id="seo-content"> was pre-rendered but got wiped by empty() during SPA hydration, so AI saw an empty shell on its second revisit
Good for: the vast majority of modern web apps, but you must do an SEO content check (use curl to verify the key content is in the initial HTML)
Strategy 4: Pure SPA (Single Page Application) — citation rate near 0
Examples: Create React App, pure Vite + React, legacy Angular (without Universal), pure Vue CLI
How it works:
1. First request: the server returns only a near-empty <div id="app"></div> HTML shell
2. After the JS bundle loads → fetch API → render content
For AI crawlers: Fatal. AI gets the empty shell and can’t see any content.
Typical curl result:
<!DOCTYPE html>
<html>
<head>
<title>Your Site</title>
</head>
<body>
<div id="root"></div>
<script src="/static/js/main.abc123.js"></script>
</body>
</html>
This is all that AI sees. No product info, no blog content, no text information at all.
Why people still use pure SPA:
- The developer experience is familiar and easy to pick up
- Deployment is simple (pure static files → CDN)
- Early React/Vue tutorials all taught this
- The mistaken belief that Google can fully render JS = no problem (in reality, Google has limits too)
Fixes:
You can’t simply “upgrade to SSR” (it often requires rewriting the whole architecture), but there are three paths:
Fix A: use a prerender service
A service (Prerender.io, Rendertron) detects crawlers by User-Agent, renders dynamically, and returns the HTML.
- Pro: no rewrite needed
- Con: paid service, and may be treated as cloaking by AI vendors
Fix B: migrate to Next.js / Nuxt
Progressively migrate your existing React/Vue code to an SSR framework.
- Pro: architecturally correct for the long term
- Con: large engineering effort (1–3 months for a mid-size project)
Fix C: pre-render key content inside the SPA shell
Manually embed key SEO content inside index.html (the homepage’s service intro, FAQ, testimonials) as <noscript> or a default-visible <article>, then let the SPA decide whether to hide it after hydration.
- Pro: minimal change, immediate effect
- Con: manual maintenance, can desync
- Example: geoweb.tw uses this very technique (M3-18) —
<article id="seo-content">sits alongside<div id="app-root">, with CSS controlling visibility
A common engineering misjudgment
“Googlebot can render JS, so AI should be able to too, right?“
This inference is wrong in three places:
- Googlebot’s JS execution has a timeout (usually 5 seconds). If your hydration takes longer than 5 seconds, the content still won’t be seen
- Training crawlers (GPTBot / ClaudeBot / CCBot) run on different infrastructure than Googlebot — they don’t get a Chrome renderer
- Live-citation crawlers (ChatGPT-User / PerplexityBot) have a latency budget (the user won’t wait), so they usually only run simple JS
Conclusion: “Googlebot is OK” cannot be extrapolated to AI crawlers being OK. You have to verify them separately.
Measure: the “no-JS visible content” of every key page on your site
Run the shell script below to check multiple pages:
#!/bin/bash
URLS=(
"https://yoursite.com/"
"https://yoursite.com/products"
"https://yoursite.com/about"
"https://yoursite.com/blog"
"https://yoursite.com/contact"
)
for url in "${URLS[@]}"; do
size=$(curl -s -A "GPTBot/1.0" "$url" | wc -c)
echo "$url → ${size} bytes"
done
Health metric: each page ≥ 10,000 bytes with substantive text content.
If any page is < 5,000 bytes, that page effectively doesn’t exist as far as AI is concerned.
An easily overlooked strategy: llms.txt
No matter which rendering strategy you use, you can create a /llms.txt as a plain-text fallback:
# Your Company
Your Company is [one-sentence intro], focused on [service area].
## Main Services
- Service A: [short description]
- Service B: [short description]
## Contact
Email: [email protected]
Website: https://yoursite.com/
AI crawlers especially like this file (it has become a de facto standard), and even if your main site is a JS-heavy SPA, at least /llms.txt gives AI a reliable source of facts.
For the detailed spec, see the previous article: GPTBot / ClaudeBot / PerplexityBot — Rule Differences and Best Settings for the 8 Major AI Crawlers
Step one: use curl to verify your site’s visibility to AI
Run the curl commands from this article’s “quick verification” section to see whether your site is SSR/SSG/Hybrid/pure SPA for GPTBot.
If it’s a pure SPA: your GEO investment ROI will be near 0, and no matter how much content you write, it’s wasted. Solve the rendering problem first.
👉 Run the GeoWeb health check for a full 12-dimension assessment — the health check’s “HTML capacity” and “semantic structure” dimensions will reveal the SPA shell problem.
Migrating rendering strategy is an engineering-grade project (1–3 months for a mid-size project), not something to figure out as you go. Framework selection, a progressive migration plan, performance baseline comparisons, AI-crawler verification, cross-team coordination (frontend / backend / DevOps / SEO) — getting any one of these decisions wrong even once can take a site out of service for months. GEO managed service includes an engineering consulting assessment, sparing companies from digging the pit themselves: [email protected]
GEO deep-dive series. Previous article: “Site-Wide Cross-Page Consistency — Why Template Sameness Gets Down-Ranked by AI”