Noindex and Nosnippet for AEO: What Each Directive Does to AI Citation
Noindex and nosnippet are technical tags that tell search engines what to do with your pages. For AEO, these tags work differently for AI crawlers - a page hidden from Google search may still be fully visible to ChatGPT and Perplexity. Understanding which tag does what for AI citation is essential for controlling your content's AI visibility.
Traditional assumptions - noindex hides the page from everyone, nosnippet prevents content extraction - are no longer accurate in the multi-crawler AI environment. Each directive operates independently on each crawler type, requiring a deliberate crawl permission strategy.
For related context, see Robots.txt for AEO and AI Crawler Budget.
Crawl Directive Comparison
| Directive | Scope | Effect | AEO Impact |
|---|---|---|---|
| noindex | Page | Removes from Google index. AI crawlers still read. | Neutral - AI can still cite if crawled |
| nosnippet | Page | Prevents SERP snippet. Does NOT block AI extraction. | Reduces AI extraction, doesn't block it |
| max-snippet:N | Page | Limits snippet to N characters only. | Allows partial AI citation (first N chars) |
| GPTBot Disallow | robots.txt | Blocks OpenAI crawler. Does NOT block Google AI. | Blocks ChatGPT training + ChatGPT Search |
| Google-Extended Disallow | robots.txt | Blocks Google AI training crawler only. | Blocks Gemini training; may affect AI Overviews |
| noindex + AI Disallow | Both | Full exclusion from index AND all AI crawlers. | Complete exclusion - no AI citation possible |
Which Directive Do You Need?
Answer three questions to get the exact directive for your situation.
Do you want this page indexed by Google?
AI Crawler Robots.txt Reference
OpenAI (ChatGPT)
User-agent: GPTBot Disallow: /
Google AI Training
User-agent: Google-Extended Disallow: /
Perplexity AI
User-agent: PerplexityBot Disallow: /
Anthropic Claude
User-agent: ClaudeBot Disallow: /