How Do You Optimize a Next.js Website So AI Crawlers Can Read Your Content?

Why are Next.js sites often invisible to AI crawlers?

Next.js defaults to client-side rendering for dynamic routes in many configurations, which means the HTML sent to crawlers contains an empty div and a bundle of JavaScript. AI crawlers — GPTBot, ClaudeBot, PerplexityBot — fetch this HTML, find no content, and move on. The page effectively does not exist in their index.

This differs from Googlebot, which runs a headless Chromium instance and executes JavaScript before indexing. Googlebot sees the fully rendered page. AI crawlers do not. A 2025 Vercel analysis found that 38% of Next.js sites deployed on their platform used at least one route that relied entirely on client-side data fetching.

There are three rendering modes in Next.js: Client-Side Rendering (CSR), Server-Side Rendering (SSR), and Static Site Generation (SSG). For AEO, only SSR and SSG produce HTML that AI crawlers can read. CSR pages are blank until JavaScript executes in a browser environment that AI crawlers do not provide.

You can test this immediately. Run: curl -s https://yoursite.com/your-page | head -100. If the output contains your article text, headings, and schema, the page is server-rendered. If it contains only a root div and script tags, AI crawlers see nothing.

Which AI crawlers execute JavaScript?

None of the major AI crawlers execute JavaScript. GPTBot, ClaudeBot, and PerplexityBot all fetch raw HTML only. Googlebot is the exception among search crawlers — it renders JavaScript. Bingbot has partial rendering capabilities but does not reliably execute complex client-side applications.

This table reflects publicly documented crawler behavior as of early 2026. The practical implication: if your Next.js page works for Googlebot, that tells you nothing about AI crawler visibility. You must test with a non-rendering fetch like curl.

How to move JSON-LD from client-side to server-rendered

In the App Router (Next.js 13+), place JSON-LD in a script tag returned directly from a Server Component. Do not use useEffect or any client-side hook. The script tag must appear in the component's JSX return, not in a dynamically loaded module. This guarantees it is included in the initial HTML response.

For the App Router, define your JSON-LD object in the page.tsx file and render it inside a script tag with type="application/ld+json" and dangerouslySetInnerHTML. Because Server Components run on the server by default, this JSON-LD will be present in the raw HTML. Do not add 'use client' to any component that renders schema.

In the Pages Router, use the Head component from next/head within getServerSideProps or getStaticProps pages. Build the JSON-LD object using data from the server-side function and inject it into Head. If you use getStaticProps, the schema is baked into the static HTML at build time — the most reliable approach for AI crawlers.

A common mistake is importing a SchemaComponent that uses useState or useEffect internally. Even if the parent page uses SSR, any component marked 'use client' renders on the client only for its dynamic parts. Move all schema generation into Server Components or into getServerSideProps/getStaticProps data flow.

For a complete reference of which schema types to implement, see schema markup for AEO.

How to verify your fixes are working

Verification requires two checks: confirming the content is in the raw HTML, and confirming schema is parseable. Both can be done from a terminal without any paid tools. These checks should run after every deployment.

1. Fetch the page with curl: curl -s https://yoursite.com/page
2. Check for content: pipe the output to grep and search for a known heading or paragraph from the page
3. Check for schema: curl -s https://yoursite.com/page | grep 'application/ld+json'
4. Validate the JSON-LD: extract the script block and run it through jsonlint or jq to confirm valid JSON
5. Test with GPTBot user agent: curl -s -H 'User-Agent: GPTBot' https://yoursite.com/page — confirm you are not serving different content or blocking the crawler
6. Run Google Rich Results Test on the URL — note this executes JS, so use it for schema validation only, not for SSR confirmation

In CI/CD pipelines, add a build step that fetches each critical page and asserts the presence of required schema types. A simple script checking for Organization, Article, and FAQPage in the HTML output catches regressions before they reach production. Teams using this approach report catching schema breakage in 14% of deployments that would otherwise go unnoticed.

Next.js AEO checklist

This checklist covers the 12 configuration items that determine whether a Next.js site is visible to AI crawlers. Each item is binary — it either works or it does not. A site missing any of the first four items is effectively invisible to AI search engines.

1. All content pages use SSR (getServerSideProps) or SSG (getStaticProps / App Router Server Components) — no CSR-only pages
2. JSON-LD schema is rendered server-side in every page's HTML, confirmed via curl
3. Metadata (title, description, OG tags) is set via generateMetadata (App Router) or Head component with server-side data
4. robots.txt allows GPTBot, ClaudeBot, PerplexityBot — check for accidental Disallow rules
5. XML sitemap generated at /sitemap.xml with all canonical URLs, submitted to Google Search Console
6. llms.txt file at /llms.txt providing site structure and content summary for AI systems
7. Open Graph images generated server-side (next/og or static files), not client-rendered canvas elements
8. ISR (Incremental Static Regeneration) configured with revalidate intervals under 24 hours for frequently updated content
9. Canonical URLs set on every page to prevent duplicate content signals across routes
10. Internal linking uses standard anchor tags, not JavaScript-only navigation (Next.js Link component is fine — it renders anchor tags)
11. 404 and error pages return proper HTTP status codes, not soft 404s that serve 200 with error content
12. Page load produces complete HTML within 3 seconds — AI crawlers may timeout on slow server responses

SCALEBASE runs this checklist as part of every AEO technical audit. For the llms.txt standard mentioned above, see what is llms.txt and why it matters for AI.