Answer engine optimization: what actually moves the needle in SEO and AEO

The small set of work behind most of the result, taken only from what Google, OpenAI, Anthropic and Perplexity publish about their own systems.

· 16 min read · Nazmul Ahmed

Answer engine optimization (AEO) is the work of getting your pages used and cited in AI answers such as Google's AI Overviews and AI Mode, ChatGPT search, Claude and Perplexity. For Google, AEO is SEO: its AI features draw on the normal Search index and normal ranking, and Google says there are no extra requirements. The work that matters is, in order: pass the eligibility gates, let the non-Google search crawlers in, publish first-hand content nobody else could write, keep business data current, and measure only from first-party sources.

On this page
  1. The core fact: for Google, AEO is SEO
  2. How a page ends up in an AI answer
  3. The impact ladder
  4. Gate one: the five eligibility checks for Google
  5. Gate two: crawler access for ChatGPT, Claude and Perplexity
  6. A robots.txt starting point
  7. What to do about the crawlers
  8. The multiplier: content nobody else could have written
  9. E-E-A-T: trust first
  10. The amplifiers Google lists for AI features
  11. What to stop doing: Google's own list
  12. What the myth list does not cover
  13. What each switch actually does
  14. How to measure it without guessing
  15. The one-page checklist
  16. How nazmc.com applies it
  17. Sources
  18. Common questions

The core fact: for Google, AEO is SEO

Google's generative features do not have a separate ranking system. They retrieve pages from the ordinary Search index using ordinary ranking, then generate an answer on top. So the same levers drive classic search and AI answers, with one extra task that is genuinely AEO rather than SEO: letting the crawlers of ChatGPT, Claude and Perplexity reach your site.

From Google Search's perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.
Google Search Central, Optimizing your website for generative AI features
There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.
Google Search Central, AI features and your website

How a page ends up in an AI answer

Two mechanics decide which pages get picked: query fan-out and retrieval-augmented generation. Google's own example, step by step:

  1. The user asks a question

    "How to fix a lawn full of weeds."

    fan-out: the model issues related sub-queries

  2. Sub-queries run behind the scenes

    "Best herbicides for lawns", "remove weeds without chemicals", "prevent weeds in lawn". Because of this, a wider and more varied set of pages gets linked than in a classic result.

    core Search ranking retrieves pages for each

  3. The answer is grounded in retrieved pages

    The model reads the pages core ranking returned. Only indexed, snippet-eligible pages are candidates.

    Retrieval-augmented generation

  4. Answer plus links

    The response carries prominent, clickable links to the supporting pages. AI Overviews only appear when Google judges them additive, so for many searches they do not trigger at all.

The impact ladder

Read it from the top. The first tier is binary: miss it and nothing else counts. The second is where Google says the most leverage sits. The last two amplify and measure.

  1. Tier 1 · Gate

    Eligibility for Google

    Crawlable, returns 200, indexable, snippet-eligible, and the Search Console generative AI control set to Include. Fail any one and the page is invisible to every AI feature.

  2. Tier 1 · Gate

    Crawler access for the non-Google engines

    OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed at robots.txt and at the firewall or CDN. Blocked means not shown in that engine's search answers.

  3. Tier 2 · Multiplier

    Non-commodity, first-hand content

    Google says this will influence your presence in generative AI "more than any of the other suggestions".

  4. Tier 3 · Amplifier

    Entity data, findability and experience

    Business Profile, Merchant Center, structured data that matches the page, internal links, text-first content, supporting images and video, page experience, no duplication.

  5. Tier 4 · Instrument

    Measurement from first-party sources

    The Search Console generative AI report and GA4 referrals. The only data that is not an estimate.

Notice what the ladder does not contain: a percentage. The platforms publish an order and an emphasis, not weights. Anyone quoting a share of AI citations is using a third-party model, and Google states that no third-party tool has access to its internal ranking or AI systems.

Gate one: the five eligibility checks for Google

Five gates in series. A page has to pass every one to be a candidate for AI Overviews or AI Mode, and each can be checked in Search Console.

  1. Googlebot is not blocked

    robots.txt, login walls and CDN or hosting rules must all let Googlebot in.

    Check: Page Indexing and Crawl Stats

  2. The page returns HTTP 200

    Only pages that return a success status are indexed; error pages are not.

    Check: URL Inspection

  3. The content is indexable

    Text in a supported file type that does not break the spam policies. Blocking a URL in robots.txt is not the same as noindex, and a blocked URL can still appear.

    Check: URL Inspection

  4. The page is snippet-eligible

    nosnippet, data-nosnippet, max-snippet and noindex reduce or remove what AI features can show.

    Check: robots meta tag and X-Robots-Tag

  5. The generative AI control is set to Include

    Search Console, Settings, Search generative AI. The default is Include; Exclude removes both the links and the grounding use. Google's help page records it as rolled out to all sites on 31 August 2026.

    Check: Settings, Search generative AI

Passing every gate does not guarantee indexing or serving, and Google says so explicitly. The generative AI control is not a ranking signal for the rest of Search and has nothing to do with AI training; that is Google-Extended.

Gate two: crawler access for ChatGPT, Claude and Perplexity

Each engine runs separate bots for search, for training and for fetches a user triggers, and each is controlled on its own. Pick an engine to see its bots, from each company's own crawler documentation.

OAI-SearchBot

Purpose
Surfaces sites in ChatGPT search.
If blocked
Not shown in ChatGPT search answers; may still appear as a navigational link.
robots.txt
Honoured; changes take about 24 hours.
IP list
openai.com/searchbot.json

GPTBot

Purpose
Collects data for training foundation models.
If blocked
Content is excluded from training. Independent of search.
robots.txt
Honoured.
IP list
openai.com/gptbot.json

ChatGPT-User

Purpose
Page visits a user asks for, and GPT Actions.
If blocked
Not used to decide search inclusion.
robots.txt
May not apply, because the visit is user-initiated.
IP list
openai.com/chatgpt-user.json

The common failure is not robots.txt. It is a WAF or CDN bot-protection rule that silently blocks the search bot while robots.txt says it is welcome.

A robots.txt starting point

The search bots and the training bots are separate switches. This lets the three AI search engines in and leaves training as a decision you make on its own:

robots.txt
# AI search: let these engines find and cite your pages
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Model training is a separate choice.
# To opt out of it, uncomment these lines:
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /

Keep your existing Disallow rules for private paths (admin, checkout, account) in every group; allowing a bot is not the same as allowing it everywhere. Then do the half robots.txt cannot do: allow the same bots at your WAF or CDN.

What to do about the crawlers

Watch out

  • Blocking Anthropic by IP aloneIt stops the bot reading your robots.txt, so an opt-out may not persist.
  • Assuming a disallow hides a pageA page disallowed to OAI-SearchBot can still surface as a link and title if OpenAI learns the URL elsewhere. Keeping it out takes noindex, and the crawler has to be allowed in to read the tag.
  • Relying on robots.txt for user fetchersChatGPT-User and Perplexity-User may ignore it, because a person asked for the page.

Do this

  • Allow the three search bots in robots.txtOAI-SearchBot, Claude-SearchBot and PerplexityBot.
  • Allow them at the WAF or CDN tooBy user agent and by published IP range. Perplexity documents Cloudflare and AWS WAF rules; OpenAI asks you to confirm your host allows its search bot's IP addresses.
  • Refresh the IP lists on a scheduleEach company publishes its ranges as a JSON feed, and they change.
  • Decide training separatelyYou can allow OAI-SearchBot and disallow GPTBot; the two are independent.

The multiplier: content nobody else could have written

Google's guide says non-commodity content will matter more than any of its other suggestions. Its own contrast shows what that means:

Commodity

  • "7 Tips for First-Time Homebuyers"Common knowledge anyone could write, or a generative model could produce. Recycling what others have already said.
  • Page-per-variation productionA page for every query variation or fan-out query is scaled content abuse under the spam policies, and Google calls it an ineffective long-term strategy.
  • Writing to a word count, re-dating for freshnessGoogle says it has no preferred length. Mass-producing across topics without expertise, or AI content made mainly to manipulate rankings, is spam.

Non-commodity

  • "Why We Waived the Inspection and Saved Money: A Look Inside the Sewer Line"An experienced take that goes beyond the ordinary.
  • Your own point of view and dataFirst-hand reviews, original research, reporting or analysis that adds substantial value compared with the other results.
  • A clear Who, How and WhyA byline that leads to the author's background; the method and evidence, including AI assistance where a reader would expect it; made for people, not for search visits.

For a service business, the non-commodity asset is usually its own work: the jobs, the cases, the pricing logic and the mistakes. Generic "what is X" explainers are the commodity end of the scale unless they carry something first-hand.

E-E-A-T: trust first

E-E-A-T (experience, expertise, authoritativeness and trust) is not a single ranking factor, but Google's systems use a mix of signals that identify it, and Google says trust is the most important part. Topics that affect health, money, safety or society get extra weight. Some of the self-assessment questions Google publishes:

  • Would you expect to see this content in a printed magazine, an encyclopedia or a book?
  • Is it written or reviewed by someone who demonstrably knows the topic?
  • Does it show first-hand expertise, such as actually using the product or visiting the place?
  • Will a reader leave feeling they have learned enough, without needing to search again?

The practical reading: fewer, deeper pages with named authors beat many thin ones. Google says a high page count does not make a site higher quality.

The amplifiers Google lists for AI features

None of these is new. Google's page on AI features lists them as the SEO fundamentals that remain worthwhile for AI Overviews and AI Mode.

Entity data

Business Profile and Merchant Center

AI answers can include local business information and product listings drawn from them. Keep both accurate and current.

Structured data

Match it to the visible text

Not required for AI features, and there is no special schema to add. Keep using it for rich results, and make sure it says what the page says.

Findability

Internal links, text first

Make pages reachable through internal links, and keep important content in text, not only inside images, video or script-rendered widgets.

Experience

Page experience

Displays well on every device, loads quickly, main content easy to find. Google rewards page experience across many aspects, not one or two.

Hygiene

Less duplication, JavaScript done right

Duplicates waste crawl resources. JavaScript content is fine if it is not blocked; semantic HTML helps, but perfect code is not required.

Agents

An interface agents can read

Browser agents read the page structure and accessibility tree. OpenAI says its Atlas browser interprets ARIA roles and labels, so accessible markup helps agents too.

Add images and video that support the text as well: AI features can surface them, which gives a page more ways to appear.

What to stop doing: Google's own list

Google published a list of things site owners do not need to do for its AI features. It is where much of the effort sold as AEO or GEO goes.

Ignore for Google

  • llms.txt, special AI markup, Markdown copiesGoogle Search does not use them; they will neither harm nor help. Fine to keep for other systems.
  • Chunking content into tiny piecesNo such requirement. Google understands several topics on one page, and there is no ideal length.
  • Rewriting content "for AI"The systems understand synonyms and meaning; you do not need a sentence for every long-tail phrasing.
  • Chasing inauthentic mentionsCore ranking rewards quality and other systems block spam, and AI features depend on both.
  • Over-focusing on structured dataNot required, and there is no special schema for AI features.
  • Tools claiming Google's internal metricsGoogle says no third-party tool has access to its ranking or AI systems.

Do instead

  • Fix the gates firstCrawl access including the CDN and firewall, 200s, indexability, snippet controls, the generative AI control. Then verify in Search Console.
  • Let the three non-Google search bots inAt robots.txt and at the firewall, by user agent and IP range.
  • Publish fewer, first-hand, bylined piecesWith Who, How and Why answered on the page.
  • Keep Business Profile and Merchant Center currentThey feed local and product answers directly.
  • Measure from Search Console and GA4Generative AI impressions by page, and ChatGPT referrals by their utm_source.

What each switch actually does

Teams routinely flip the wrong switch. Pick a control to see its documented effect:

robots.txt for Googlebot

Affects
Whether Google crawls the page for Search, and so for every AI feature.
Does not affect
Whether the URL can still appear: a blocked URL can show without its content.

How to measure it without guessing

Two first-party sources exist. Everything else is a third-party estimate.

Google

Search Console generative AI report

Impressions from AI Overviews and AI Mode by page, country, date and device; recorded as rolled out to all sites worldwide on 31 August 2026. The chart counts by property (two links from one site in one answer are one impression), the page tab by page, with the usual 1,000-row limits.

Google

Clicks in the normal Performance report

Clicks from AI features are counted under the Web search type. Google reports that clicks from results with AI Overviews are higher quality, with users tending to spend more time on the site.

ChatGPT

Referrals in GA4

ChatGPT adds utm_source=chatgpt.com to the links it sends, so its search traffic can be tracked in GA4 or any analytics tool.

Claude and Perplexity

Referrals and server logs only

Both document their bots and IP ranges but publish no performance report for site owners. Visibility shows up in referrals and server logs.

Treat any "share of AI answers" or "AI citation percentage" figure from an SEO tool as a modelled estimate, and label it that way in reports.

The one-page checklist

Work through it in order. Tick items as you go; your progress stays in this browser.

0 of 10 done

How nazmc.com applies it

This site follows the same list. Its robots.txt allows the crawlers that can cite it, including OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot, and blocks only Common Crawl's training crawler. Every article in this section opens with a direct answer to its own question, written so an answer engine can lift it without the rest of the page, and carries a named author and a real publication date.

If you want the same work done on your own site as part of a wider marketing programme, the digital marketing services page sets out how that is scoped. For the broader picture of where AI fits in a marketing team, see what AI marketing is and how it is used.

Sources

Primary sources only, compiled on 11 September 2026. No third-party studies or SEO-tool metrics are used as evidence.

  1. 1Google Search CentralOptimizing your website for generative AI features on Google Search
  2. 2Google Search CentralAI features and your websiteLast updated 10 December 2025.
  3. 3Google Search CentralGoogle Search technical requirementsLast updated 18 December 2025.
  4. 4Google Search CentralCreating helpful, reliable, people-first content
  5. 5Google Search Console HelpGenerative AI performance report (Search)
  6. 6Google Search Console HelpSearch generative AI control
  7. 7OpenAIOverview of OpenAI Crawlers
  8. 8OpenAI Help CenterPublishers and Developers FAQ
  9. 9Anthropic, Claude Help CenterDoes Anthropic crawl data from the web, and how can site owners block the crawler?7 April 2026.
  10. 10PerplexityPerplexity Crawlers

Common questions

What is answer engine optimization?
Answer engine optimization (AEO) is the work of getting your pages used and cited in AI-generated answers, such as Google's AI Overviews and AI Mode, ChatGPT search, Claude and Perplexity. For Google it is the same work as SEO; for the other engines it adds one task, letting their search crawlers reach your site.
Is AEO different from SEO?
For Google, no. Google says its AI features retrieve from the normal Search index using normal ranking and have no additional requirements. The genuinely separate AEO task is crawler access: allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot at robots.txt and at the firewall.
Do I need an llms.txt file for AI search?
Not for Google. Google says llms.txt and special AI markup will neither harm nor help in Search. Other AI companies do not publish an equivalent statement, so keeping one for other systems is harmless, but it should not be the priority.
If I block GPTBot, will my site disappear from ChatGPT?
No. GPTBot collects training data and is independent of search. ChatGPT search uses OAI-SearchBot, so you can allow OAI-SearchBot and disallow GPTBot. The same split applies to Anthropic, where ClaudeBot trains and Claude-SearchBot serves search.
How do I measure whether AI search sends me traffic?
Use first-party data. Search Console's generative AI performance report shows AI Overview and AI Mode impressions by page, and ChatGPT adds utm_source=chatgpt.com to its referral links, so those visits show in GA4. Treat tool-reported AI citation percentages as estimates.
What matters most for appearing in AI Overviews?
First, the gates: the page must be crawlable, return 200, be indexable and be snippet-eligible, with the Search Console generative AI control left on Include. After that, Google says non-commodity, first-hand content will matter more than any of its other suggestions.
See the digital marketing services