Short answer
Separate search discovery, model training and user-triggered fetching. Audit robots.txt and firewall rules before making a crawler-by-crawler decision.
The answer in detail
Separate AI search discovery from model training before changing robots.txt. Search bots, training crawlers and user-triggered fetchers do different jobs. Blocking one can affect visibility while blocking another expresses a training preference. A blanket “block AI” switch hides those differences.
The benchmark
The source groups the major published agents as follows. Names and behavior are product documentation, so review them before deployment rather than treating this table as permanent.
| Purpose | OpenAI | Anthropic | Perplexity | |
|---|---|---|---|---|
| Search discovery | OAI-SearchBot | Claude-SearchBot | PerplexityBot | Googlebot plus relevant Search controls |
| Model-related use | GPTBot | ClaudeBot | No training bot declared in the cited guide | Google-Extended control token |
| User-triggered fetching | ChatGPT-User | Claude-User | Perplexity-User | Different product-specific fetchers |
The source cites OpenAI's distinction between OAI-SearchBot and GPTBot: blocking the training crawler is separate from exclusion from ChatGPT search. Blocking search access can prevent inclusion of page content in search answers, while navigational links may still appear. The source reports roughly 24 hours for a change to be reflected; do not interpret that as a guaranteed service level.
Anthropic warns that blocking its search or user agents can reduce visibility. Perplexity says its search bot is not used for training and that user-requested fetching generally does not follow robots.txt in the same way as automated crawling. A file intended for cooperative crawlers is not access control.
Google-Extended is a robots control token rather than a separate HTTP crawler. Google's documentation separates its Gemini-related training and grounding scope from Google Search crawling and ranking. A separate Search Console generative-AI control governs inclusion in AI Overviews, AI Mode and generative features in Discover. Current Google guidance says that control rolled out worldwide on August 31, 2026.
The rule
Choose by use and business objective. A store wanting accurate product citations usually needs its product information accessible to relevant discovery systems. It may still choose to restrict training use. Put that decision in writing so a later firewall change does not silently reverse it.
Accessibility does not guarantee a recommendation, ranking or sale. If your own pages cannot be fetched, a reseller's accessible page may become the easier source, but that is a risk to investigate rather than a predictable outcome for every query.
Evaluate Google-Extended separately because its scope includes Gemini-related uses beyond training. Likewise, the Search Console inclusion control does not override all other product participation choices, including Merchant Center and Ads. Use the official controls for the actual surface you want to affect.
The steps
- Fetch the live
/robots.txton your domain. Inspect named groups and the wildcard group, then test the exact URL paths that matter. A firewall, CDN or security service can block requests even when robots.txt permits them. - Check policy pages. The source identifies
/policies/in Shopify's default rules and describes a fragrance brand whose team could not find the rule in its custom template because it came from the defaults. Verify the live file. If delivery and return terms are otherwise inaccessible, use Shopify's documented customization approach or provide accurate accessible policy content. - Preserve Shopify's default Liquid generation when editing
robots.txt.liquid. Back up the existing template and review the generated output. The platform's documentation warns that an incorrect edit can damage search access. - If your decision is to restrict training, add deliberate groups for the appropriate tokens. The source's example pairs
User-agent: GPTBotwithDisallow: /, and separately pairsUser-agent: ClaudeBotwithDisallow: /. Do not copy a wildcard block in their place. Add Google-Extended only after accepting its distinct scope. - Inspect Search Console > Settings > Search generative AI. Confirm whether the property includes content, excludes it or inherits its parent's setting. Choose intentionally rather than assuming a training setting controls this surface.
- Recheck server responses and bot logs after changes. Then assess brand citations over a consistent set of queries after an appropriate observation period. Visibility changes alone do not isolate the effect of the file.
The pitfall
The most common harmful block can be accidental: an overbroad wildcard, an SEO application rewriting the file or a security setting blocking every AI request. Diagnose the layer responsible before editing a second system.
Robots.txt also does not protect private content. Use authentication and proper authorization for that. User-triggered fetchers and noncooperative agents may behave differently from the crawlers listed here, and user-agent strings can be impersonated. Finally, llms.txt does not replace crawl controls.
What to tell your team
“We will keep the discovery access our business needs and document training preferences separately. The source's fictional 1,200 monthly ChatGPT visits illustrate why we should test the right crawler rather than use a global AI block. Actual traffic impact still needs measurement.”
