Technical guide
AI crawlers: who they are and what to allow
AI companies send three kinds of visitors: search crawlers that index pages so their assistant can cite you, fetchers that read a page because a user asked, and training crawlers that collect content for future models. Each is controlled separately in robots.txt. Blocking the search crawlers can remove you from that assistant’s answers; blocking training crawlers is a separate business decision.
The three kinds of AI visitor
| Company | Search / indexing | User-triggered fetch | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | — |
| Googlebot (also feeds AI Overviews and AI Mode) | — | Google-Extended (a robots.txt token, not a separate crawler) | |
| Apple | Applebot | — | Applebot-Extended (token) |
| Microsoft | Bingbot (also feeds Copilot) | — | — |
| Meta | — | — | Meta-ExternalAgent |
| Common Crawl | — | — | CCBot (widely used in training datasets) |
Vendors document different robots.txt behaviour for user-triggered fetchers, and names change. Check each vendor’s current documentation before acting; third-party lists disagree on details.
A sensible default for a business that wants to be found
- Allow Googlebot and Bingbot: they feed Google’s and Microsoft’s AI answers.
- Allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to be cited in those assistants.
- Decide on training crawlers separately. Blocking GPTBot, for example, does not by itself remove you from ChatGPT search, which uses OAI-SearchBot.
- Keep private paths (admin, APIs, carts) disallowed for everyone.
- Remember robots.txt controls crawling, not indexing; use noindex for pages that must stay out.
Check your own file
Our free AI crawler checker reads a robots.txt you paste in and shows which of these agents are allowed or blocked from your home page. Nothing is fetched or stored.
Related services
Sources
Primary references this guide relies on. Provider documentation changes, so check the source before acting.
Questions
OpenAI documents GPTBot for training and OAI-SearchBot for search as separate controls, so blocking GPTBot alone does not remove a site from ChatGPT search.
Ready to talk through your project?
Send a voice note, spreadsheet or a plain description of where your business is struggling. We’ll help you find the right next step.

