How do AI answer engines decide which sites to cite?

They retrieve pages from a search index, then quote the passages that answer the question most clearly. So the work has two parts: get into the indexes these systems read, and publish passages worth quoting.

Google describes its AI Overviews and AI Mode as built on retrieval-augmented generation and “query fan-out” from its core Search index, and says in its guide to generative AI features that optimizing for them is still SEO. Bing’s Webmaster Guidelines say Bing and Copilot rely on the same crawling, indexing and ranking foundation. ChatGPT, Perplexity and Claude run their own search crawlers as well. No engine promises inclusion: Bing states that optimization “does not guarantee grounding or citations.”

Can AI crawlers read your pages?

Only if the content is in the HTML. Serve important text, prices, FAQs and contact details as server-rendered or pre-rendered HTML, not content assembled by JavaScript after load or hidden behind tabs and modals.

Google can process JavaScript but calls JavaScript sites “generally more complex” for SEO. Bing’s guidelines tell site owners to avoid “hiding critical content behind client-side rendering” and warn that content that cannot be reliably rendered may not be selected for grounding. A quick test: view the page source (not the inspector) and confirm your key answers are there.

Which AI crawlers should robots.txt allow?

Allow the search crawlers you want citations from, and decide separately about training crawlers. As of October 2026, the companies document these user agents:

Company User agent Purpose (per the company) If you block it
OpenAI OAI-SearchBot Surfaces sites in ChatGPT search Not shown in ChatGPT search answers, though links may still appear
OpenAI GPTBot Collects content that may train models Signals content should not be used for training
OpenAI ChatGPT-User Fetches pages when a user asks robots.txt rules may not apply (user-initiated)
Perplexity PerplexityBot Surfaces and links sites in Perplexity search; not used for training Perplexity recommends allowing it to appear in search results
Perplexity Perplexity-User Fetches pages for a user’s question Generally ignores robots.txt (user-initiated)
Anthropic Claude-SearchBot Indexes content to improve Claude’s search results May reduce visibility in Claude’s search results
Anthropic ClaudeBot Collects content that may train models Future content excluded from training
Anthropic Claude-User Fetches pages when a Claude user asks May reduce visibility for user-directed search
Google Google-Extended Robots.txt token for Gemini training and grounding No effect on Google Search inclusion or ranking

Sources: OpenAI’s crawler overview, Perplexity’s crawler docs, Anthropic’s crawler article and Google’s common crawlers page.

Three practical notes. OpenAI and Perplexity both say robots.txt changes can take about 24 hours to take effect. Google’s AI features page confirms that AI Overviews and AI Mode are governed by Googlebot’s rules and snippet controls (nosnippet, max-snippet, noindex), not by Google-Extended. And check your CDN or firewall: a bot-protection setting can block these crawlers even when robots.txt allows them.

Why does Bing matter for Copilot?

Copilot is grounded in Bing’s index, so a page Bing hasn’t indexed is unlikely to be cited there. Bing’s guidelines cover Bing search, Copilot and “grounding API results,” and its AI Performance report covers Copilot, AI summaries in Bing and “select partner AI integrations.”

  1. Verify the site in Bing Webmaster Tools and submit your XML sitemaps, with accurate lastmod values.
  2. Set up IndexNow: host a key file, then notify on every publish, update and deletion. Per the IndexNow FAQ, one submission is shared with all participating engines, including Bing, Yandex, Naver, Seznam.cz and Yep.
  3. Avoid NOARCHIVE on pages you want cited. Bing says it prevents content from being used in Copilot responses, and NOCACHE limits Copilot to the URL, title and snippet.
  4. Check indexing status in Webmaster Tools after each significant release.

What does Google say about AI Overviews and AI Mode?

Google says there are no special requirements beyond being indexed and eligible for a snippet. Its generative AI guide adds that a site “must be included in Search generative AI features in Search Console,” lists things you can ignore for Google Search (llms.txt, “chunking” content, rewriting for AI, inauthentic mentions), and recommends “non-commodity” content: first-hand experience and expert takes rather than restated common knowledge.

How should pages be written so they get quoted?

Answer first, with attributes. Open each section with a one or two sentence answer that makes sense pasted alone, then support it. Bing’s guidelines ask for the same: explicit facts, key information early, and content understandable “without external context.”

  • Attributes, not adjectives. “Same-day service in Travis and Williamson counties, $89 call-out fee” can be quoted. “Fast, affordable, trusted” cannot.
  • Consistent entity naming. Use the same business name, description and service names on every page, in schema and in your listings.
  • One page per buyer constraint. Give genuinely different situations their own page: a budget limit, a regulated category, a business size, a deadline. Don’t spin pages for wording variants. The same rule applies to landing pages for paid traffic. Google warns that creating content for every query variation to manipulate AI responses violates its scaled content abuse policy.
  • Comparison content. Buyers ask assistants to compare options. A fair comparison table (your product against named alternatives, with criteria, prices and where each fits best) answers that directly. Bing’s AI Performance report classifies grounding queries by intent, and Comparison is one of its categories.

Which structured data helps?

Structured data that describes the visible page accurately. Google’s structured data guidelines require markup to be “a true representation of the page content,” and Bing’s say misleading markup may be ignored. Neither engine requires special schema for AI answers.

  • Organization on every page, with name, logo, contact details and sameAs links. Google describes sameAs as a page on another site with more information about your organization, such as a social or review profile.
  • Article for guides and posts, with real dates.
  • BreadcrumbList to describe site structure.
  • FAQPage only where the questions and answers are visible. Google stopped showing FAQ rich results in May 2026, per its documentation updates, so the markup describes the page but won’t produce that result.

Do reviews, directories and community mentions matter?

They matter because assistants read what others say about you, not just what you say. Keep your listings on review platforms, industry directories and business profiles accurate and consistent with your site, and link them via sameAs. Take part in communities where your buyers ask questions, as yourself and with disclosure. Google notes that seeking inauthentic mentions “isn’t as helpful as it might seem,” and its spam systems apply to AI responses too.

Is llms.txt worth adding?

It’s a cheap convenience, not a ranking factor. llms.txt is a proposal for a Markdown file at your site root that summarizes the site and links to clean content for AI tools. Google says Search ignores these files and they neither help nor hurt. If you add one, generate it from your real pages so it can’t drift.

How do you measure AI visibility?

Treat AI answer engines as their own channel, tied to leads and revenue like any other (see our measurement service), and use each platform’s own data where it exists.

Signal Where to find it
ChatGPT referrals OpenAI’s publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs
Other AI referrals Referrers such as perplexity.ai, copilot.microsoft.com and gemini.google.com, grouped into one AI channel in analytics
Copilot and Bing citations Bing Webmaster Tools AI Performance: citations, cited pages and grounding queries (sampled; citations are not clicks)
Google AI features Search Console’s Generative AI performance report, per Google’s guide
Crawler activity Server or CDN logs filtered by user agent

Filter crawler logs to 200 responses and verify the source. Scanners spoof AI user agents, so a user-agent match alone proves nothing. OpenAI, Perplexity and Anthropic publish IP lists, and Google documents how to verify its crawlers by reverse DNS or published ranges.

How MSG approaches this

MSG runs SEO and AEO as one program and measures them separately. We start with a diagnosis: crawler access, rendering, Google and Bing indexing, IndexNow and structured data that matches the page. Then we build answer-first content from the questions buyers actually ask, with consistent entity data across the site and its listings. AI referrals and citations get their own line in reporting next to paid. We say what we know and what we don’t: no one can promise rankings or citations, so we improve the inputs and report what happens. See our SEO and AEO service.

Sources

  1. Google Search Central: Optimizing your website for generative AI features on Google Search
  2. Google Search Central: AI features and your website
  3. Google Search Central: Google's common crawlers (Google-Extended)
  4. Google Search Central: General structured data guidelines
  5. Google Search Central: Organization structured data
  6. Google Search Central: Latest documentation updates
  7. Google Search Central: Verifying Googlebot and other Google crawlers
  8. OpenAI: Overview of OpenAI crawlers
  9. OpenAI Help Center: Publishers and developers FAQ
  10. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  11. Perplexity: Perplexity crawlers
  12. Bing Webmaster Guidelines
  13. Bing Webmaster Tools: AI Performance
  14. IndexNow: FAQ
  15. Bing: IndexNow getting started
  16. schema.org: sameAs
  17. llms.txt proposal

Platform features and policies change. This guide reflects the sources above as of October 7, 2026.