One Page, Two Readers: SEO vs. AI Answer Engines
Your page now has two audiences—Google's search index and AI answer engines—and they don't read it the same way.
Your website has always had one reader to please: Google's crawler. That's no longer true. Every page you publish is now also read by a second, more fragmented set of bots — OpenAI's, Perplexity's, and others — that decide whether your business gets mentioned in a ChatGPT or Perplexity answer. These two readers don't process your page the same way, and building for one doesn't automatically satisfy the other. This post breaks down how each one actually works, where they disagree, and what that means for a studio or business publishing a page today.
How the search engine reader works
Google's process has three stages: crawling, indexing, and ranking. Crawling is Googlebot discovering and downloading a page's text, images and video. Indexing is Google analyzing that content, working out relevance and canonical status, and storing it in its database (Google Search Central).
The important catch: being crawled is not automatic. Google itself says the vast majority of pages are found and added because a crawler stumbled onto them while exploring the web — not because someone submitted them (Google Search Central).
If your site leans on JavaScript to render content, there's a second wrinkle. Google's crawling happens in two passes: first it fetches the raw HTML and queues links it finds, and only once resources allow does a headless Chromium instance actually render the page and execute the JavaScript, after which Googlebot parses the rendered HTML for any links it missed the first time (Google Search Central, JavaScript SEO basics). Content that only appears after a script runs can be delayed or simply missed if it isn't implemented carefully. This is a common failure point for single-page apps and heavily animated marketing sites built without server-side rendering.
On top of the raw HTML, structured data acts as connective tissue. Schema.org JSON-IPD markup is a standardized vocabulary that lets you mark up your content so Google, Microsoft, Yahoo and Yandex can all understand it consistently (Semrush, schema markup guide).
How the AI answer engine reader works
AI systems don't use one crawler with one job — they use several, each doing something distinct. OpenAI is explicit about this split: OAI-SearchBot and GPTBot are controlled by separate robots.txt tags, so a site can allow OAI-SearchBot to appear in search results while blocking GPTBot from using that same content to train a generative model (OpenAI crawler documentation).
Perplexity mirrors this structure. PerplexityBot does general web crawling and builds a retrieval index Perplexity draws on for answers — it is explicitly not used to train foundation models — while a separate agent fetches pages live at the moment a user asks a question (Perplexity crawler docs).
This separation has produced a real economic standoff. As of August 2024, 35.7% of the world's top 1,000 websites were blocking GPTBot, a seven-fold jump from the 5% blocking rate when the crawler launched a year earlier (icoda.io, GPTBot guide), largely because AI crawlers consume enormous volumes of content while sending back very little referral traffic in return (ppc.land, OpenAI crawler policy analysis). Despite that, blocking hasn't slowed growth: GPTBot's share of AI crawling traffic rose from 4.7% to 11.7% between July 2024 and July 2025 (icoda.io).
Google's own AI layer behaves differently again — it mostly reuses the existing search index rather than building a new one. Google states there are no additional technical requirements to appear in AI Overviews or AI Mode; a page just needs to be indexed and eligible to show a normal snippet (Google Search Central, AI features). But Google Overviews can use a "query fan-out" technique, issuing multiple related searches across subtopics and data sources to build one response (Search Engine Land, AI Overviews guide). One user question can quietly trigger dozens of mini-searches across your different pages, not a single lookup against your homepage.
Side-by-side: what each reader actually needs
| Search engine (Google) | AI answer engines | |
|---|---|---|
| Crawler | Googlebot | GPTBot, OAI-SearchBot, PerplexityBot, others |
| Discovery | Automatic, via links found while crawling (source) | Fragmented — each bot has its own job and its own robots.txt directive (source) |
| JS rendering | Crawl-then-render, two-pass process (source) | No public documentation from Perplexity on whether it renders JavaScript or how strictly it enforces robots.txt |
| Schema markup | Used as a shared vocabulary across Google, Microsoft, Yahoo, Yandex (source) | Google says it is not required for AI Overviews/AI Mode citation (source) |
| Extra requirements to be cited | Indexed + snippet-eligible (source) | None confirmed beyond crawlability; llms.txt proposal exists but isn't confirmed to be used |
Where the sources disagree
Three specific questions don't have a settled answer yet, and it's worth being honest about that instead of pretending there's a clean playbook.
Does schema markup help you get cited by AI? Google says directly that you do not need to add schema markup to be visible in AI responses, and there is no special schema.org markup required for AI Overviews or AI Mode (Google Search Central). But the same guidance hedges: schema gives search engines explicit information about entities, which may reduce ambiguity, though whether it does the same job for AI systems is unclear, and adding it doesn't guarantee retrieval or citation (same source). SEO vendors keep recommending it anyway, as general hygiene rather than a proven lever (Semrush).
Does llms.txt actually do anything? The proposal describes a small markdown file at a domain's root that gives AI systems a curated map of a site's most important content (Search Engine Land; Semrush). An independent server-log test tells a different story: from mid-August to late October 2025, a site's llms.txt file received zero visits from Google-Extended, GPTBot, PerplexityBot, or ClaudeBot (Semrush). If your team is being pitched llms.txt as a must-have, treat it as speculative, not proven.
Do AI crawlers respect robots.txt reliably? Perplexity documents PerplexityBot as respecting standard crawl directives and recommends allowing it. But independent trackers note Perplexity has never published whether its crawler renders JavaScript or how strictly it enforces exclusions — a real documentation gap compared with Google's far more detailed public rendering process (51degrees, Perplexity AI 2026).
A practical checklist for one page, two readers
Given the above, here's a reasonable set of things to check on any page you're about to publish or rebuild:
- Server-render or pre-render critical content. Don't rely on client-side JavaScript alone for text you want indexed or cited — Google's rendering pass is a second, resource-gated step, not guaranteed to happen quickly (source).
- Decide deliberately on GPTBot, OAI-SearchBot and PerplexityBot in robots.txt. These are independent settings — you can allow indexing in search while blocking your content from AI model training, or vice versa (source).
- Keep JSON-LD schema markup on your pages regardless. It's confirmed useful for four major search engines and is plausible, if unconfirmed, for AI systems (Semrush; Google).
- Don't treat llms.txt as a guaranteed win. It's a reasonable low-cost addition, but real crawler logs show major bots ignoring it as of late 2025 (Semrush).
- Write for query fan-out, not just a single keyword. Since AI Overviews can spin one question into multiple sub-searches, cover a topic's adjacent subtopics on the same page or in linked pages rather than optimizing for one narrow phrase (Search Engine Land).
- Get basic indexing right first. Google is blunt that appearing in AI Overviews or AI Mode requires nothing beyond being indexed and snippet-eligible — there's no shortcut that skips ordinary SEO fundamentals (source).
A note on India-specific rules
There isn't one. None of the primary documentation from Google Search Central, OpenAI, Perplexity or schema.org publishes India-specific thresholds or rules. The crawling and rendering mechanics described here apply identically whether your site is hosted in Pune or Portland. If a vendor tells you there's a separate "India ruleset" for AI visibility, that claim isn't backed by any of the primary sources reviewed here — treat the technical requirements above (crawlable HTML, renderable JavaScript, valid schema, deliberate robots.txt choices) as universal, not local.
Questions people also ask
Should I block GPTBot to stop AI from using my content?
That's a business decision, not a technical one. OpenAI lets you block GPTBot (training) while still allowing OAI-SearchBot (search visibility) independently (source). Blocking has become common — over a third of top-1000 sites did it by August 2024 (icoda.io) — but it hasn't stopped GPTBot's crawling traffic share from growing (icoda.io), so blocking is more about content-use policy than visibility control.
Do I need to rebuild my JavaScript-heavy site to appear in AI answers?
You need it to be crawlable and renderable for Google's index first, since AI Overviews and AI Mode draw on pages that are already indexed and snippet-eligible (Google Search Central). If your JavaScript content isn't reliably rendered in Google's two-pass crawl process, it won't reliably reach either reader (source).
Is adding llms.txt worth the effort right now?
It's low-cost to add, but there's no confirmed payoff yet. A server-log test found zero visits to an llms.txt file from Google-Extended, GPTBot, PerplexityBot or ClaudeBot over a two-and-a-half month window in late 2025 (Semrush). Add it if you like, but don't delay more proven fundamentals — indexing, schema, renderable content — waiting for it to matter.